Looking to reduce compile time.

lambchops511

Senior member
Apr 12, 2005
659
0
0
Im currently using Lynnfield 2.8 GHz (i5 760)

Any ideas when the 6-core ivy bridge chip will be out? Or should I get the 6-core sandy bridge or 4-core ivy bridge?
 

WhoBeDaPlaya

Diamond Member
Sep 15, 2000
7,414
402
126
If you're willing to pay the price premium for X79, go with the hex core (or more - dual procs, etc.)
If not, probably IVB (if you have a Microcenter nearby).

Replacing my work-provided workstation (decrepid E6600 w/3GB RAM) with my own 3570K @ 4.2GHz, 32GB RAM and SSD certainly increased my productivity. Or it would have, if I didn't spend the "spare" time reading /. :sneaky:
 

Subyman

Moderator <br> VC&G Forum
Mar 18, 2005
7,876
32
86
What are you compiling and what program are you using?
 

Smoblikat

Diamond Member
Nov 19, 2011
5,184
107
106
If you want to be a true compiling master, grab a quad opteron board and shove some 16 or 24 core optys in there.

Otherwise grab a SB-E 3930K.
 

lambchops511

Senior member
Apr 12, 2005
659
0
0
What are you compiling and what program are you using?

Custom program. Only able to parallelize around 10 files at a time due to dependencies. But if I save 10 mins a day, that is 60 hours in a year. If I value my time at $20 / hr (which I value more), thats around 1k. And yes, I work 365 days in a year.
 

inf64

Diamond Member
Mar 11, 2011
3,884
4,690
136
I'd say get 3930K.12 threads @ 3.2Ghz and unlocked. You can probably push it close to 4Ghz on air and it should cut your compile time a lot. The downside is the price, ~550USD+whatever the board is.
 
Last edited:

NTMBK

Lifer
Nov 14, 2011
10,398
5,629
136
I'd say get 3930K.12 threads @ 3.2Ghz and unlocked. You can probably push it close to 4Ghz on air and it should cut your compile time a lot. The downside is the price, ~550USD+whatever the board is.

Agreed. The Sandy Bridge E chips have quad channel memory too, which will help with feeding your cores. You can get more cores with the Xeon chips (up to 16 cores/32 threads in a dual socket workstation), but you'll get lower clock speeds and since you can only parallelise 10-wide those threads won't make a bit of difference. Overclocked 6 core and fast quad-channel RAM is the way to go.

The other piece of the puzzle is disk access- you might want to move to an SSD (or at the very least a high performance hard drive).
 

tulx

Senior member
Jul 12, 2011
257
2
71
The FX 8120 and 8150 are great value (better price/performance than SB in heavily multi-threaded tasks). AMD recently reduced the prices even more. You can get a 8120 for around 150 USD now.
 

Pilum

Member
Aug 27, 2012
182
3
81
The FX 8120 and 8150 are great value (better price/performance than SB in heavily multi-threaded tasks). AMD recently reduced the prices even more. You can get a 8120 for around 150 USD now.
Actually the FX CPUs are likely bad value in this context. It's about saving time by using the highest-end hardware available, time which is worth several thousand dollars over an assumed 3-year lifecycle for the machine. A $400 difference for CPU+MB is insignificant here. If you save $400 dollars on initial investment, but loose time worth $600 over the next three years, you actually take a loss of $200. Welcome to the wonderful world of economic analysis.

@lambchops511: yeah, 3930K should be the best bet. The 8-core Xeons would be better, but they start at $1000+. Might be better to overclock the 3930K slightly if you're comfortable with that.
 

NTMBK

Lifer
Nov 14, 2011
10,398
5,629
136
Ignore people suggesting the Opterons. You need strong single threaded performance for compilation- if you have a dependency chain in your build and wind up waiting on one or two especially large CPP files to finish compiling before the build system can start building the next bit, then it doesn't matter how many threads you have sitting spare. (I speak from experience here- it's frustrating as hell when my 12 core workstation has 22 threads sat idle, while 2 cores are maxed out for minutes at a time.)
 

lambchops511

Senior member
Apr 12, 2005
659
0
0
Ignore people suggesting the Opterons. You need strong single threaded performance for compilation- if you have a dependency chain in your build and wind up waiting on one or two especially large CPP files to finish compiling before the build system can start building the next bit, then it doesn't matter how many threads you have sitting spare. (I speak from experience here- it's frustrating as hell when my 12 core workstation has 22 threads sat idle, while 2 cores are maxed out for minutes at a time.)

Absolutely true. Which is why I can't get good parallelization more than approx 6 threads depending on the stage of compilation. Also the linking stage (which is very long) is single threaded.
 
Last edited:

lambchops511

Senior member
Apr 12, 2005
659
0
0
Agreed. The Sandy Bridge E chips have quad channel memory too, which will help with feeding your cores. You can get more cores with the Xeon chips (up to 16 cores/32 threads in a dual socket workstation), but you'll get lower clock speeds and since you can only parallelise 10-wide those threads won't make a bit of difference. Overclocked 6 core and fast quad-channel RAM is the way to go.

The other piece of the puzzle is disk access- you might want to move to an SSD (or at the very least a high performance hard drive).

Thanks. Does extra memory bandwidth help compile? I thought the object files were small enough to fit in cache. Also already on SSD.
 

zephyrprime

Diamond Member
Feb 18, 2001
7,512
2
81
Thanks. Does extra memory bandwidth help compile? I thought the object files were small enough to fit in cache. Also already on SSD.

Compiling doesn't take a lot of memory. It's definitely not throughput computing. Might want to set up a ram drive though? Big L2 or L3 cache disproportionately affects performance.
 

sm625

Diamond Member
May 6, 2011
8,172
137
106
Maybe you should try overclocking your i5-760. You might get the best performance by disabling two cores and punching the remaining two up to 3.5, and setting single core turbo up around 4GHz. i5-760 stock clocks are downright miserable. They left a LOT of performance on the table.
 

samboy

Senior member
Aug 17, 2002
223
94
101
I do a lot of compiling and went with a 2600k@4.5GHZ.
This is one area where hyper threading actually helps (around 50%).

I purchased my setup 12+ months before the 6 core chips were out.
Not really worth my time to upgrade since the extra 2 cores don't seem to make too much extra difference according to the compile benchmarks:-

http://www.anandtech.com/bench/CPU/334

Benefit from Non-Hyperthreading to Hyper is 8 seconds
Benefit from 4 Core to 6 Core gives another 4 seconds
(Note: The relative gain I saw was similar to these results)

If I was building from scratch then you may as well get the 6 core and overclock.
 

Idontcare

Elite Member
Oct 10, 1999
21,110
59
91
Absolutely true. Which is why I can't get good parallelization more than approx 6 threads depending on the stage of compilation. Also the linking stage (which is very long) is single threaded.

When you compile, are you only running one job at a time or do you load up your computer with more than one compile job (multitask the compiling efforts of many jobs?).

The parallelization within a given job may not be all that stellar, but you can parallelize the workload itself by loading up multiple independent jobs in parallel. If this is the case for you then going with a sea-of-cores will likely be better for you in the end.

If you are only working on one project at a time then this approach doesn't help you, naturally, and you are better off going with a CPU that has great single-threaded performance and really aggressive turbo-clocking to minimize the time spent processing serial portions of the job.

(I'm assuming you aren't overclocking since you mentioned your time is worth money and it would be rather foolish to OC your CPU by 20% with the risk of trashing all the output and having to double your time spent on the job to get it right the second time through)
 

Phynaz

Lifer
Mar 13, 2006
10,140
819
126
I'd say get 3930K.12 threads @ 3.2Ghz and unlocked. You can probably push it close to 4Ghz on air and it should cut your compile time a lot. The downside is the price, ~550USD+whatever the board is.

You don't overclock when you are making a living from your CPU.

Oops, IDC beat me to it.
 

lambchops511

Senior member
Apr 12, 2005
659
0
0
When you compile, are you only running one job at a time or do you load up your computer with more than one compile job (multitask the compiling efforts of many jobs?).

The parallelization within a given job may not be all that stellar, but you can parallelize the workload itself by loading up multiple independent jobs in parallel. If this is the case for you then going with a sea-of-cores will likely be better for you in the end.

If you are only working on one project at a time then this approach doesn't help you, naturally, and you are better off going with a CPU that has great single-threaded performance and really aggressive turbo-clocking to minimize the time spent processing serial portions of the job.

(I'm assuming you aren't overclocking since you mentioned your time is worth money and it would be rather foolish to OC your CPU by 20% with the risk of trashing all the output and having to double your time spent on the job to get it right the second time through)

Actually my entire project has only something like 20 files to compile, however some files generates temporary files which need a second pass for compile, but it takes around 30 secs to compile the entire project. I literally do hundreds of compiles a day, so if I reduce this compile time by 50%, it really adds up in terms of time value money.

And no, I will not (and currently do not) OC. I only buy Intel Mobos too, yes they cost $20 more, but I also rather not spend extra time googling around how to get XXX hardware working together with YYY hardware. I pay that extra $20 and believe Intel did that extra grunt work for hardware validation.
 

NP Complete

Member
Jul 16, 2010
57
0
0
You don't mention what OS your tool chain is on?

If you're on Windows, double check that your build directories are excluded from your anti-virus scanning - when I was working on firmware at Intel, adding our build directory to the AV exclusion list gained us about 20-30% performance boost.

My other suggestion is to use performance monitoring tools to characterize what are your current bottlenecks - compile toolchains can have different bottlenecks depending on the tools you're using, and how the compiler is written. Check your memory, disk IO, and CPU utilization.

In my current job, we have a mix of tools that run on each build. Some, like the compiler seem to be constrained either CPU, or disk IO (largely based on how many dependencies a particular object has). Others, like our static analysis tools are primarily constrained by CPU.

Also check how many concurrent builds your toolchain allows - some build tools (gmake, etc) allow you to specify a flag to concurrently build things (though
this can cause issues if your dependencies aren't good). It's always nice to get a free boost if you can turn on concurrent builds

Good luck!
 

Idontcare

Elite Member
Oct 10, 1999
21,110
59
91
Actually my entire project has only something like 20 files to compile, however some files generates temporary files which need a second pass for compile, but it takes around 30 secs to compile the entire project. I literally do hundreds of compiles a day, so if I reduce this compile time by 50%, it really adds up in terms of time value money.

And no, I will not (and currently do not) OC. I only buy Intel Mobos too, yes they cost $20 more, but I also rather not spend extra time googling around how to get XXX hardware working together with YYY hardware. I pay that extra $20 and believe Intel did that extra grunt work for hardware validation.

How large are your source files? You will no doubt benefit from having a disk subsystem that has excellent IOPs (I did/do).

A modern SSD is fast for large IOPS with small file changes:



But an SSD on a raid card which has cache (2GB minimum) can work even better:



However if your total files are small (< 6GBs or so), then a software based ramdrive will be unbeatable for improving your compile times:

 

lambchops511

Senior member
Apr 12, 2005
659
0
0
You don't mention what OS your tool chain is on?

If you're on Windows, double check that your build directories are excluded from your anti-virus scanning - when I was working on firmware at Intel, adding our build directory to the AV exclusion list gained us about 20-30% performance boost.

My other suggestion is to use performance monitoring tools to characterize what are your current bottlenecks - compile toolchains can have different bottlenecks depending on the tools you're using, and how the compiler is written. Check your memory, disk IO, and CPU utilization.

In my current job, we have a mix of tools that run on each build. Some, like the compiler seem to be constrained either CPU, or disk IO (largely based on how many dependencies a particular object has). Others, like our static analysis tools are primarily constrained by CPU.

Also check how many concurrent builds your toolchain allows - some build tools (gmake, etc) allow you to specify a flag to concurrently build things (though
this can cause issues if your dependencies aren't good). It's always nice to get a free boost if you can turn on concurrent builds

Good luck!

Linux. Mixture of gcc and a few other compilers / static analysis for other things that arent cpp.
 

Soulkeeper

Diamond Member
Nov 23, 2001
6,731
155
106
I setup a 12GB ram drive (it only uses what I put in it) and often compile code (untar/compress, transcode, grep, etc.) there
although I don't always notice the difference i'm sure it's faster than compiling on my SSD. It takes about 1 second to copy most anything to the ram drive to compile it.
I have a source code dir over 100GB on my 1TBx2 raid1 incase I want to reference it later

tons of cache, fast cores, low latency/fast mem, SSD FTW
compiling is truley the best system stability test
 
Last edited:
sale-70-410-exam    | Exam-200-125-pdf    | we-sale-70-410-exam    | hot-sale-70-410-exam    | Latest-exam-700-603-Dumps    | Dumps-98-363-exams-date    | Certs-200-125-date    | Dumps-300-075-exams-date    | hot-sale-book-C8010-726-book    | Hot-Sale-200-310-Exam    | Exam-Description-200-310-dumps?    | hot-sale-book-200-125-book    | Latest-Updated-300-209-Exam    | Dumps-210-260-exams-date    | Download-200-125-Exam-PDF    | Exam-Description-300-101-dumps    | Certs-300-101-date    | Hot-Sale-300-075-Exam    | Latest-exam-200-125-Dumps    | Exam-Description-200-125-dumps    | Latest-Updated-300-075-Exam    | hot-sale-book-210-260-book    | Dumps-200-901-exams-date    | Certs-200-901-date    | Latest-exam-1Z0-062-Dumps    | Hot-Sale-1Z0-062-Exam    | Certs-CSSLP-date    | 100%-Pass-70-383-Exams    | Latest-JN0-360-real-exam-questions    | 100%-Pass-4A0-100-Real-Exam-Questions    | Dumps-300-135-exams-date    | Passed-200-105-Tech-Exams    | Latest-Updated-200-310-Exam    | Download-300-070-Exam-PDF    | Hot-Sale-JN0-360-Exam    | 100%-Pass-JN0-360-Exams    | 100%-Pass-JN0-360-Real-Exam-Questions    | Dumps-JN0-360-exams-date    | Exam-Description-1Z0-876-dumps    | Latest-exam-1Z0-876-Dumps    | Dumps-HPE0-Y53-exams-date    | 2017-Latest-HPE0-Y53-Exam    | 100%-Pass-HPE0-Y53-Real-Exam-Questions    | Pass-4A0-100-Exam    | Latest-4A0-100-Questions    | Dumps-98-365-exams-date    | 2017-Latest-98-365-Exam    | 100%-Pass-VCS-254-Exams    | 2017-Latest-VCS-273-Exam    | Dumps-200-355-exams-date    | 2017-Latest-300-320-Exam    | Pass-300-101-Exam    | 100%-Pass-300-115-Exams    |
http://www.portvapes.co.uk/    | http://www.portvapes.co.uk/    |