Build advice needed

[daniel1926]

I’m going to be doing a project where I set up and test a few open source web crawlers and search engines over the next month. I’ve fooled around with Open Search Server on my laptop (Intel Haswell i5 1.3Ghz, 8GB, Mac Air) and my poor little machine is completely tapped out.

I would like to build a Linux machine to run all of these tests. As the bulk of the time in web crawling is spent waiting for data, the more threads the merrier. I would like to spend about $500 on this project but could stretch a little bit higher if it would result in markedly better performance. I would prefer to buy off of eBay/craigslist as this is a one time project for me and I’d like to keep the overall costs down.

What motherboards/cpus /memory do you guys recommend for a project like this one? I don’t mind two or three year old technology if it can do what I need it to do (in fact I would prefer it as it will be at a discount to 2014 product). I’d like a mid sized case as I am not sure my wife will appreciate a full tower case hanging around the apartment. I do not play computer games so I do not think I will need a graphics card. Should I use a small SSD for system software and then a multi terabyte HDD for data? Or will it work just fine if I keep everything on a single HDD?

I am planning on testing a few engines that are based on vector space models, but other than Apache Nuch/Solr, do you know of any other search engine packages that use an implementation of the PageRank algorithm or some other link analysis algorithm to judge relevancy of search results? The vector space model may have worked well when the web was small, but it is giving me very poor results today.

Answers for the questions in the sticky:

450 to 500 dollar budget

I will be buying parts in the United States

I have no brand preference. I just need the system to work as intended

I do not have any parts that I will be using

I definitely will not be overclocking

I will be using an old monitor that I will borrow from a friend. Not sure what its resolution is. It was a standard issue office/work monitor from about 5 years ago.

I will build it in the next couple of weeks

I’m planning on using open source software for everything. No Windows or MS Office etc.

 

[mfenn]

As the bulk of the time in web crawling is spent waiting for data, the more threads the merrier.

To be clear, threads which are waiting for data do not use CPU resources. If you're bound by the latency of the network and remote server, then you don't need a 1:1 mapping of software threads to hardware threads.

Usually the only way to get discounts on older hardware that make sense from a performance point of view is to look at used enterprise gear. However, most of those servers are going to be pretty loud.

Speaking of data, how much storage space do you expect to need? A big HDD will eat a good chunk of your budget right off the bat, so it's important to plan for that.

 

[daniel1926]

Total data will top out at just under 1.5 terabytes.

If there aren't any good discounts on used processors, then what is a good new one to get? Top end speed really doesn't matter in my use case. Anything, say, 2.5 Ghz or faster will be more than enough.

 

[Burpo]

http://www.ebay.com/itm/Dell-Inspir...097?pt=LH_DefaultDomain_0&hash=item35e2842289

Condition of the item/Manufacturer Refurbished,

What's in the box: Desktop, Power cord, Mouse, Keyboard, Recovery DVD

Processor: Intel® Core™ i5-4440 Processor (6M Cache, 3.1GHz, up to 3.30 GHz with turbo boost)
Operating System: Windows 8.1 64 bit Home
Graphics: HD 4400
Memory: 8 GB Dual Channel DDR3 at 1600 Mhz
Hard Drive: 1TB
Optical Drive: DVD RW
Network Card/Ports: ( WIFI) Wireless card,
2 USB 3.0 (back), 4 USB 2.0 (front, back), 8-in-1 Media Card Reader, Mic and Headphone Jacks, HDMI, ,RJ-45,

Warranty: DELL hardware warranty till December 2015

 

[daniel1926]

Great! That fits the bill. If I wanted to add more ram, what should I get?

 

[mfenn]

http://www.ebay.com/itm/Dell-Inspir...097?pt=LH_DefaultDomain_0&hash=item35e2842289

Condition of the item/Manufacturer Refurbished,

What's in the box: Desktop, Power cord, Mouse, Keyboard, Recovery DVD

Processor: Intel® Core™ i5-4440 Processor (6M Cache, 3.1GHz, up to 3.30 GHz with turbo boost)
Operating System: Windows 8.1 64 bit Home
Graphics: HD 4400
Memory: 8 GB Dual Channel DDR3 at 1600 Mhz
Hard Drive: 1TB
Optical Drive: DVD RW
Network Card/Ports: ( WIFI) Wireless card,
2 USB 3.0 (back), 4 USB 2.0 (front, back), 8-in-1 Media Card Reader, Mic and Headphone Jacks, HDMI, ,RJ-45,

Warranty: DELL hardware warranty till December 2015

:thumbsup: Nice find!

OP, the PC above has 2 DIMM slots, so you can get a 2x8GB kit like this G.Skill for $114.

 

[daniel1926]

Thanks guys! This is really helpful.