Thinking out loud - render farm

[mnewsham]

Please don't tell me that's network cabling on top...

And a Sharpie! :awe:

 

[Idontcare]

Please don't tell me that's network cabling on top...

:sneaky: sure...ah...it's not network cable...mostly :sneaky:

At least it wasn't as bad as this: (not mine)

 

[taltamir]

Would using a thin client network boot via PXE be a practical alternative to giving each CPU a HDD?

 

[PreferLinux]

Would using a thin client network boot via PXE be a practical alternative to giving each CPU a HDD?

Skip the "thin client" part anyway. A thin client does all its processing on the host/server and doesn't really do any itself.

 

[taltamir]

Skip the "thin client" part anyway. A thin client does all its processing on the host/server and doesn't really do any itself.

thanks for clarifying. Ok then, just a network boot. Would that be practical?

 

[Arkadrel]

:sneaky: sure...ah...it's not network cable...mostly :sneaky:

At least it wasn't as bad as this: (not mine)

*shock*

Wow... just... wow.... I bet thats fun when theres a issue, and you have to find the cable to replace, or the unit, and/or if you have to untangle them wires to get to the problem, and then put it all back together again.

*ps: I feel bad for the fellow that worked there <.< and was in charge of keeping things running.

 

[Idontcare]

thanks for clarifying. Ok then, just a network boot. Would that be practical?

Nope. Not for the applications we are talking about, bandwidth contention is problematic because you need your network fabric to be an available resource for scaling purposes.

Also think of the advantages in SSD's over HDD's. Going from a local HDD to a remote HDD accessed by the network would be a step in the wrong direction.

The improvement to be had would be in going from HDD to SSD. The programs create scratch files that are tens, sometimes hundreds, of gigabytes in size.

Imagine trying to run Photoshop on your personal computer but having the scratch file be located on a network drive. That would "counter productive" to say the least. Now try and have 6 or more computers (and 4 parallel instances of Photoshop on each computer) trying to access the same remote disk array simultaneously...are they each going to get 200MB/s throughput?

And even if they did, what bandwidth would left for actual interprocessor communications?

Turns out for my apps even the cheapest lowliest $40 dedicated hard-drive is a superior price/performance solution compared to any kind of remote storage alternative.

When I built the P4 cluster at TI the local IT group fought with me regarding local storage, they wanted diskless nodes (they were responsible with maintaining the system) so I had them work with DELL to run my apps on a pre-configured cluster with and without local disk storage. The performance penalty, even when as optimized as the local IT and DELL server guys could get it, was staggering.

Nearly 10x decrease in performance for the same server price (they doubled the network backbone, etc, spending as much on the sub-system as the HDD's themselves, trying to close the gap). It was a QED, I got my local drives on the cluster.

I'm sure there are some apps out there that don't suffer a performance penalty while maintaining a storage footprint that fits within the local ram so you can get away with using ramdrive and pushing the drive image on startup. But it is always a cost/performance tradeoff. Sure I could build my nodes to have enough ram that a hard-drive is irrelevant, but I'd spend so much on ram that I'd be looking at reducing my overall node count from say 12 nodes to 6 nodes.

That tradeoff works in favor of price/performance for some apps, not mine though, at least not at the current pricepoints. Maybe in another 4-6yrs time the landscape will have changed.

 

[Martimus]

I just saw this picture of a Dell Poweredge C5125 on JFAMDs blog, and I have to admit that I am impressed at how dense they are able to make that system.

I think that is 148 cores, if I am not mistaken. I wonder if that would work well for your application IDC? (Although it may be well out of your pricerange as well.)

 

[taltamir]

Nope. Not for the applications we are talking about, bandwidth contention is problematic because you need your network fabric to be an available resource for scaling purposes.

Also think of the advantages in SSD's over HDD's. Going from a local HDD to a remote HDD accessed by the network would be a step in the wrong direction.

The improvement to be had would be in going from HDD to SSD. The programs create scratch files that are tens, sometimes hundreds, of gigabytes in size.

Imagine trying to run Photoshop on your personal computer but having the scratch file be located on a network drive. That would "counter productive" to say the least. Now try and have 6 or more computers (and 4 parallel instances of Photoshop on each computer) trying to access the same remote disk array simultaneously...are they each going to get 200MB/s throughput?

And even if they did, what bandwidth would left for actual interprocessor communications?

Turns out for my apps even the cheapest lowliest $40 dedicated hard-drive is a superior price/performance solution compared to any kind of remote storage alternative.

When I built the P4 cluster at TI the local IT group fought with me regarding local storage, they wanted diskless nodes (they were responsible with maintaining the system) so I had them work with DELL to run my apps on a pre-configured cluster with and without local disk storage. The performance penalty, even when as optimized as the local IT and DELL server guys could get it, was staggering.

Nearly 10x decrease in performance for the same server price (they doubled the network backbone, etc, spending as much on the sub-system as the HDD's themselves, trying to close the gap). It was a QED, I got my local drives on the cluster.

I'm sure there are some apps out there that don't suffer a performance penalty while maintaining a storage footprint that fits within the local ram so you can get away with using ramdrive and pushing the drive image on startup. But it is always a cost/performance tradeoff. Sure I could build my nodes to have enough ram that a hard-drive is irrelevant, but I'd spend so much on ram that I'd be looking at reducing my overall node count from say 12 nodes to 6 nodes.

That tradeoff works in favor of price/performance for some apps, not mine though, at least not at the current pricepoints. Maybe in another 4-6yrs time the landscape will have changed.

Thank you for this detailed explanation.
Would large amounts of RAM negate the need to use a HDD as scratch? (which, as you said, introduced horrible delays over network).
About how many GB of ram do you think would be needed for render farm?

 

[Idontcare]

I just saw this picture of a Dell Poweredge C5125 on JFAMDs blog, and I have to admit that I am impressed at how dense they are able to make that system.


I think that is 148 cores, if I am not mistaken. I wonder if that would work well for your application IDC? (Although it may be well out of your pricerange as well.)

The photoshot of the blade layout is rather sexy

Of what I could find on the C5125 it is limited to nothing more powerful than an single X4 per blade (no thubans or magny cours) so you'd be looking at 48 cores for a 12-blade config.

I couldn't find anything on pricing, but I didn't look past the third page of search results in google. It may be too early.

Unless you are really space-limited it is a very hard sell to make the case for rack-mount anything (let alone high-density rackmount) versus just cobbling together a cluster based on COTS (commodity off the shelf) parts.

With AMD's thuban and X4 prices and the mobo prices being a notch lower than Intel's at most of the points on the performance curve, you can put together a rather ferocious computing farm for a couple thousand bucks.

If you've been building rigs since the dawn of the x86-based PC it can feel absurd at times nowadays just how much compute power your measly $400 will buy you.

In fact, since one of my apps is a windows-only app, by far the two biggest contributors to the expense of my clusters is the windows OS (nearing $170-$180 per node if you want 64bit + remote desktop capabilities) and the back-up power supply (usually around $100/node after shipping is factored).

But its really no surprise. Just look at the profit margins of Microsoft and APC versus those of AMD, any mobo maker, any dram maker, any HDD maker, and any PSU maker.

 

[Idontcare]

Thank you for this detailed explanation.
Would large amounts of RAM negate the need to use a HDD as scratch? (which, as you said, introduced horrible delays over network).

Yes it would, but the trade-off is rarely a net benefit in the price/performance dept for the overall cluster.

(which is why even modern server blades still have those HDD's on-board)

It was buried in my wall of text, I don't blame you for not catching it:

Sure I could build my nodes to have enough ram that a hard-drive is irrelevant, but I'd spend so much on ram that I'd be looking at reducing my overall node count from say 12 nodes to 6 nodes.

That tradeoff works in favor of price/performance for some apps, not mine though, at least not at the current pricepoints. Maybe in another 4-6yrs time the landscape will have changed.

About how many GB of ram do you think would be needed for render farm?

I would need approximately 8GB per thread. On a hex-core node that would mean 48GB of ram.

Performance would certainly improve by virtue of accessing a ram-drive versus HDD, but realistically we'd be talking about maybe a 10-15% performance improvement while doubling (if not more) the cost per node.

For my handful of apps the trade-off doesn't work in favor of a ram-drive based node. Maybe in a few years when $/GB drops another order of magnitude though.

 

[Zstream]

What OS was used to manage the cluster?

 

[Idontcare]

What OS was used to manage the cluster?

When I could get away with it just being a computational chemistry and rendering (POVRAY) cluster the OS was just redhat linux with their builtin management tools.

Now that I am forced to use windows owing to my current priority application (MetaTrader4) all the job queuing is managed through simple dos shell commands. The program is self-contained in that it manages itself when it comes to all matters of job spawning, results collection, data analysis, and uploading to the recipient apps for on-the-fly parameter updating.

It is far from ideal, at the same time its not holding me back either and the price is acceptable. So until something in that dynamic changes I am less likely to invest in changing it versus investing in improving other areas of the business. (from the bigger picture angle of the overall business objectives)

 

[WhoBeDaPlaya]

At least it wasn't as bad as this: (not mine)

Holy flying spaghetti monster!

 

[taltamir]

Holy flying spaghetti monster!

thats actually his cousin, Ethernet cable monster.

 

[Cerb]

For these apps, with COTS parts, how much value, if any, is there for the option of ECC RAM?

 

[wrosecrans]

Would using a thin client network boot via PXE be a practical alternative to giving each CPU a HDD?

In many cases, yes. Our farm is 64 procs (256 cores), and we have all the nodes booted over the network. (We are using CentOS, which is basically RHEL.) The nodes are mostly used for Mental Ray and some stuff like fluid sims, so we find that this arrangement is more than adequate. We have something like 24-32 GB or RAM per machine, which is good for our workloads. The oldest group nodes in our farm is about two - three years old now, so we would probably spec a new batch of nodes with at least 64 GB per box, depending on core count, to give ourselves a little headroom. You need a good storage server to feed this kind of a setup. We have an Isilon cluster, plus a SAN, plus some file servers, so it works for us.

In some workloads, large fast local storage is important. For us, it's never been an issue. Not worrying about drives failing, and the effort overhead of maintaining OS's on separate machine makes net-booting 100% win. Building a cluster / farm is all about understanding your workloads. There's not a universal right answer.

Personally, I would skip the cost of hard drives in each node, and put that budget into good power supplies. If you have enough work to justify a renderfarm, you probably have enough to keep that farm busy. Cheap consumer PSU's seem to hate being on 24x7 for months at a time cramped into some closet with a few dozen of their very warm brethren. If the PSU dies, it can kill pretty much everything in the node. (And most "serious" rackmount systems used for pro clusters will be at least dual-corded, so they can throw a PSU and keep working.)

Another thing to think about is Lights-Out-Management. You can probably set up whatever queuing software you use to do "power on the node" as the first step in a render task that it assigns to the node. You can also configure things so that a machine will shut itself down after some period of idleness. We don't do that, but for a home farm where you don't have a bunch of artists feeding it, you can reasonably expect some idle-times. You probably don't want to be paying for powering the farm 100% of the time.

 

[Martimus]

The photoshot of the blade layout is rather sexy

Of what I could find on the C5125 it is limited to nothing more powerful than an single X4 per blade (no thubans or magny cours) so you'd be looking at 48 cores for a 12-blade config.

I couldn't find anything on pricing, but I didn't look past the third page of search results in google. It may be too early.

Unless you are really space-limited it is a very hard sell to make the case for rack-mount anything (let alone high-density rackmount) versus just cobbling together a cluster based on COTS (commodity off the shelf) parts.

With AMD's thuban and X4 prices and the mobo prices being a notch lower than Intel's at most of the points on the performance curve, you can put together a rather ferocious computing farm for a couple thousand bucks.

If you've been building rigs since the dawn of the x86-based PC it can feel absurd at times nowadays just how much compute power your measly $400 will buy you.

In fact, since one of my apps is a windows-only app, by far the two biggest contributors to the expense of my clusters is the windows OS (nearing $170-$180 per node if you want 64bit + remote desktop capabilities) and the back-up power supply (usually around $100/node after shipping is factored).

But its really no surprise. Just look at the profit margins of Microsoft and APC versus those of AMD, any mobo maker, any dram maker, any HDD maker, and any PSU maker.

Yeah, it can be amazing, but I just impulse bought a new computer for about $2,600 about 15 minutes ago. I am not usually impulsive, but when I procrastinate a long time for something (like I have obviously done here - look at my signature) I can get suddenly impulsive in getting it done. The problem is that I only really researched 2/3 of the components, but luckily they were the most expensive ones.

I hope it wasn't a mistake on my part, but oh well, the damage is done.

EDIT: Oh yeah, I forgot the reason I posted it here. I meant to show that a few thousand could still be used on a single setup just as easily as it could be used on multiple cheap setups.

 

[VirtualLarry]

I just impulse bought a new computer for about $2,600 about 15 minutes ago.

No, no, no. Not $2600 - the hot new thing is the Core i7 2600K.


(Joke)

 

[aigomorla]

At least it wasn't as bad as this: (not mine)

doesnt that guy know, if you put "do not do something", the first thing one wants to do, is try and find out why?


:biggrin:

 

[Martimus]

No, no, no. Not $2600 - the hot new thing is the Core i7 2600K.


(Joke)

Even funnier is that is the processor that I bought.

Although the monitor was the most expensive part.

EDIT:This is what I bought:
HAF-X Case
Seasonic X760 PSU
Intel i7-2600K CPU
ASROCK P67 Extreme4 MB
Crucial C300 128GB SSD
Western Digital Black 2GB HD
12X LG BluRay burner
HIS HD6970 Video Card
Corsair H60 CPU Heatsink
HP ZR24w 24" Monitor
SAITEK Cyborg V5 Keyboard
16GB (4x4GB) G.Skill 1600 DDR3 RAM (1.5V, CAS 9)
Windows 7 Home Premium (64bit)
Trendnet TEW-623PI Wireless N PCI adapter
Trendnet TEW-652BRP Wireless N Router

I am sure I screwed up in there somewhere, but hopefully it is a good total system.

 

[Martimus]

If that place had laid me off, I think the last thing I would do before walking out the door would be to swap a few of those wires around. I wonder how long it would take for them to fix that? :biggrin:

 

[Schmide]

thats actually his cousin, Ethernet cable monster.

Psst those are audio cables.

I wonder how long it would take for them to fix that? :biggrin:

Depends mostly on whether the audio tech in peaking or crashing.

 

[Idontcare]

Even funnier is that is the processor that I bought.

Although the monitor was the most expensive part.

EDIT:This is what I bought:
HAF-X Case
Seasonic X760 PSU
Intel i7-2600K CPU
ASROCK P67 Extreme4 MB
Crucial C300 128GB SSD
Western Digital Black 2GB HD
12X LG BluRay burner
HIS HD6970 Video Card
Corsair H60 CPU Heatsink
HP ZR24w 24" Monitor
SAITEK Cyborg V5 Keyboard
16GB (4x4GB) G.Skill 1600 DDR3 RAM (1.5V, CAS 9)
Windows 7 Home Premium (64bit)
Trendnet TEW-623PI Wireless N PCI adapter
Trendnet TEW-652BRP Wireless N Router

I am sure I screwed up in there somewhere, but hopefully it is a good total system.

Congrats on the new gear :thumbsup:

I feel the impluse too That hardware looks solid to me! I don't see anything I would regret owning.

Mmmmmm C300....

 

[Martimus]

Congrats on the new gear :thumbsup:

I feel the impluse too That hardware looks solid to me! I don't see anything I would regret owning.

Mmmmmm C300....

Thanks,

I hope it works well, and I do a much better job at cable management with this build than my last one. I did an alright job with the computer I built for my friend a couple years ago, so I know I can do it if I try.

My current computer was built in the spring of 2004, but I have since replaced many of the components. The case is literally falling apart now though, from moving so often and the switch to turn the thing on had to be jury rigged to work about a half dozen times so far, with this last time being the most elaborate fix. (The first time, I just took it off and fixed it in the electronics lab I was working at, but I don't have that option anymore)