Sharing 40tb Multimedia data to 30 clients, one or many computers for server?

[denywinarto]

Hi, i plan to share multimedia files (around 40tb) to 30 clients from ubuntu server.
I have 2 options.
1. 1 server with asrock extreme 6 pro motherboard (10 SATA), i3 haswell, 4TB x 10 green WD, 8 gb ram and 700 watt power supply
2. 3 servers with minimum specs, probably dual core, each with 4x 4tb 4x 4tb and 2x 4tb

Now my concern with 3 computers is the electrical consumption (it's not cheap in my country) and the maintenance cost which in this case is tripled compared to the first option (except the HDD)

But with single server, my concern if, what if all clients access the file simultaneously?
Would it be enough to handle them?
I need the clients to be able to play 1080p movies smoothly btw...
I'm afraid if the transfer rate slows down it will cause movie lags

I can't find similar case on internet so i hope i can get help here..

thanks before.

 

[IndyColtsFan]

A couple of things:

1. If these servers are critical to your business, you should buy a pre-built server from Dell or HP for the support, part availability, etc.
2. If these servers are critical to your business, you would likely want to have 2 or more nodes clustered.

 

[Squeetard]

Eff that asrock junk. Get a server and a san. With 10gbe

 

[Genx87]

Get something professional if these are clients.

 

[denywinarto]

Eff that asrock junk. Get a server and a san. With 10gbe

A couple of things:

1. If these servers are critical to your business, you should buy a pre-built server from Dell or HP for the support, part availability, etc.
2. If these servers are critical to your business, you would likely want to have 2 or more nodes clustered.

Get something professional if these are clients.

Is asrock that bad?
Well the prebuilt server is too expensive..

What's the risk if i use this?
It's only for movie sharing btw,
I'm not planning to overclock it or anything
Edit : i forgot to add it's for local sharing.. LAN sharing

 

[azazel1024]

What is the usage scenario? I see 30 clients, I don't see possible simultaneous connections. Is this read only? Read/write? lock contention possible? So on.

If this is pretty much just 30 people watching 1080p movies, it should probably be fine, though if you have 30 simultaneous users, it might choke depending on the setup for those drives and the files.

BR rips at around 30-50Mbps with 30 users, if its all from a single drive where the media file(s) is located, its going to choke. You'd need a RAID array, and possibly a pretty good one to handle that many users and that much bitrate streaming off of it.

2 users, same file, BR rip, no worries. If users are accessing multiple drives (IE its just JBOD) for more than 1 file streaming all at once, no worries either (probably).

You'll also need to look at the network. If the concurrency is low, a single gigabit NIC might handle it, but if concurrency is high, you'll need at least 2 gigabit NICs in the server.

Also if concurrency is high, 8GB of RAM might be on the low side of what you might want to consider. More RAM, depending on how Ubuntu/Samba handle it (I am much more familar with Windows) might be able to take some of the pressure of the hard drives/RAID array with concurrency.

If these are lower bit rate than native BR rips, something like 8-12Mbps encodes, then its a lot more likely that whatever you have could handle 30 clients easily (even possibly/probably with 30 concurrent users).

I would not spread this across multiple servers, though I would consider a second server as a backup. If you do it across multiple servers, you are either going to have a mess of network drives for users and much higher power consumption, or if you want this to be transparent each server has duplicates, you are going to have to have a load balance router or server to balance the load across the servers...and also possibly mirroring if there are any file writes going on from users.

Why a 700w power supply? Even looking at start-up power draw, 10 drives shouldn't pull more than 150w, plus maybe 100w for the processor, motherboard and RAM. 250w max most likely. You should be fine with a much smaller power supply and that much smaller power supply is likely to be both cheaper and use less power (as you are more in to the sweet spot of efficiency, which is usually around 50% where it is the most efficient). I'd look at a 400-500w 80+ Gold PSU if you are concerned about power efficiency here.

Depending on what is up with everything, I'd look at a back up for that data, either as a warm standby server with periodic backups to it, or else a couple of USB/eSATA enclosures hooked up to the server pulling periodic backups. Thats a lot of data to possibly lose.

 

[imagoon]

As mentioned above, you have to tell use your use case. 30 clients streaming 40TB doesn't really tell us much.

30 full bluray rips at the same time streaming at 1080p is far different than 30 people with 3 simultaneous 12mbps videos. Are they watching the same thing, if so then you might need to multicast which adds network gear. If not, than you have to deal with random read. If all 30 people hit files on 1 of those 4TB drives, even 12mbps files are likely to crush it since random IO floors on disks. +NICS + network gear that can handle the load... etc

I mean our 1500 user Exchange SAN with "only" 20TB of disk was 25k due to IO load to give you an idea of what you might be looking at.

 

[Cerb]

Is asrock that bad?

Not Asrock, but using consumer/client PC hardware where it's not meant to be. That is assuming that this system going down will cost you.

 

[JackMDS]

If you do not have the right funds, and can not afford the electrical bill for the right hardware then it Not in your destiny to be able to provide Good Steaming service to 30 people.

That said miracle can happen.

Do what you plan to do with the hardware that you have and spend some time praying.

Who knows under certain circumstances (depending on the users habits) it might be that it will work well for you.

 

[azazel1024]

As mentioned above, you have to tell use your use case. 30 clients streaming 40TB doesn't really tell us much.

30 full bluray rips at the same time streaming at 1080p is far different than 30 people with 3 simultaneous 12mbps videos. Are they watching the same thing, if so then you might need to multicast which adds network gear. If not, than you have to deal with random read. If all 30 people hit files on 1 of those 4TB drives, even 12mbps files are likely to crush it since random IO floors on disks. +NICS + network gear that can handle the load... etc

I mean our 1500 user Exchange SAN with "only" 20TB of disk was 25k due to IO load to give you an idea of what you might be looking at.

I don't know how Ubuntu/Samba handles it, but from my modest Win XP/7/8/8.1 and SMB1/2/2.1/3 experience with streaming off a network share (and some of it depends on the media player too!), it is random I/O, but its large block I/O. Its generally not pulling 4k blocks off the drive, or even TCP frame size (1500-9k+). Its pulling more than that at a time for each stream. I can't tell you exactly how much it is from my experience, but running 5 high bit rate streams concurrently, it looks like the drive is doing more like 1-3MB reads off the drive on each file, maybe more, and loading it in to RAM before sticking it up through the network interface.

So it at least appears, based on disk I/O activity on my server, when doing a bunch of streams at once (just testing, usual use case for my house is 1-3 streams at a time, not the 5 or 6 I can actually do if using all possible devices), that it'll read a few MB from file A in to RAM, then a few from file B and so on, then back to file A. Rinse and repeat.

Now 30 streams at 4-6MB/sec per stream is still going to crush any hard drive though and many RAID arrays...but at least with something like a 12Mbps stream times 30 there might be a chance, especially since it should be mostly large block random I/O with multiple streams.

 

[imagoon]

I don't know how Ubuntu/Samba handles it, but from my modest Win XP/7/8/8.1 and SMB1/2/2.1/3 experience with streaming off a network share (and some of it depends on the media player too!), it is random I/O, but its large block I/O. Its generally not pulling 4k blocks off the drive, or even TCP frame size (1500-9k+). Its pulling more than that at a time for each stream. I can't tell you exactly how much it is from my experience, but running 5 high bit rate streams concurrently, it looks like the drive is doing more like 1-3MB reads off the drive on each file, maybe more, and loading it in to RAM before sticking it up through the network interface.

So it at least appears, based on disk I/O activity on my server, when doing a bunch of streams at once (just testing, usual use case for my house is 1-3 streams at a time, not the 5 or 6 I can actually do if using all possible devices), that it'll read a few MB from file A in to RAM, then a few from file B and so on, then back to file A. Rinse and repeat.

Now 30 streams at 4-6MB/sec per stream is still going to crush any hard drive though and many RAID arrays...but at least with something like a 12Mbps stream times 30 there might be a chance, especially since it should be mostly large block random I/O with multiple streams.

Pretty sure it depends on the apps and filesystem cluster sizes. A smart DLNA server will pre read because it is pretty obvious what the next read will be. Default NTFS clusters are 4KB but can be set to 64KB. So if a file is opened for random access via a drive share via SMB, Windows does make some assumptions and try to read ahead cache, however it will not do (a lot of) read ahead caching in lieu of other pending reads.

Basically "it depends" and since he hasn't really state what he is doing and what apps he is using it is hard to predict.

 

[greenhawk]

I agree with needing to know more information.

If only a Gbit network than it does not matter about the server as much as that will bottle neck everything enough to effect playback for less than 30 client PC's. Looking to setup load balancing needs multiple network cards with each PC using a different IP for the server or a network switch that can be setup to know how to do load balancing.

Going to 10Gbit networking gear can help, but you still need a pricy hardware switch to take 10Gbit in and split it up to the rest at a slower Gbit speed (assuming not also getting the faster gear for all 30 PC's as well instead of the default Gbit that most boards have now).

With 40TB of data, how it is stored will be a big issue, specifically if power usage is a concern. Spining all the drives just for one user will add up over time with a raid array. Adding more users and you start to need faster drives (good seek times) so going for "green" drives becomes an issue as well (more so if the raid implementation does not like green drives).

Given the large amount of data and the number of clients, do you expect them all to be accessing the same data about the same time or would they all be after something different depending on tastes? If the something different then splitting the load over individual drives can be a better approach as you might only have 3 people on each drive and not have to worry about speed. If everyone wants the same data within a short period of time, then a raid array or even a SSD for new/high usage files might be worth considering.

Another question is are the clients only going to be reading from the media server? no copying off from it? Any writing back to it? The copying is an issue as if one or two copy the data, it will place a higher need on speed as it will effect the ability of others to watch smoothly. Without control of individual speeds, one user can effect the rest (need some sort of software management in this case). With the coping of data to the server, it can be an issue if using a software raid as some have a very slow write time, which will effect people trying to read from the same array at the same time.

Just a point regarding setup costs as 40TB is not a small amount of data (assuming no backup). 10 of the largest most expensive drives can be an issue. But going for smaller more cost effective drives leads to issues with having them connected all at once for use. So even looking at a one off special motherboard, you might have to look at getting some sort of SAS setup for drives or a motherboard with lots of drive controllers.

As one person mentioned before though, the instant power need to power up 10+ drives at once is not a small amount. If you read some of the power specs on some drives, they will report needing 2A on the 12V rail just to start spinning. This drops off when running but still, that is nearly 25W per drive, so 250W for 10 drives and it does not count what the rest of the computer is wanting for that up to a second of time after the power switch is pressed.

 

[denywinarto]

What is the usage scenario? I see 30 clients, I don't see possible simultaneous connections. Is this read only? Read/write? lock contention possible? So on.

If this is pretty much just 30 people watching 1080p movies, it should probably be fine, though if you have 30 simultaneous users, it might choke depending on the setup for those drives and the files.

BR rips at around 30-50Mbps with 30 users, if its all from a single drive where the media file(s) is located, its going to choke. You'd need a RAID array, and possibly a pretty good one to handle that many users and that much bitrate streaming off of it.

2 users, same file, BR rip, no worries. If users are accessing multiple drives (IE its just JBOD) for more than 1 file streaming all at once, no worries either (probably).

You'll also need to look at the network. If the concurrency is low, a single gigabit NIC might handle it, but if concurrency is high, you'll need at least 2 gigabit NICs in the server.

Also if concurrency is high, 8GB of RAM might be on the low side of what you might want to consider. More RAM, depending on how Ubuntu/Samba handle it (I am much more familar with Windows) might be able to take some of the pressure of the hard drives/RAID array with concurrency.

If these are lower bit rate than native BR rips, something like 8-12Mbps encodes, then its a lot more likely that whatever you have could handle 30 clients easily (even possibly/probably with 30 concurrent users).

I would not spread this across multiple servers, though I would consider a second server as a backup. If you do it across multiple servers, you are either going to have a mess of network drives for users and much higher power consumption, or if you want this to be transparent each server has duplicates, you are going to have to have a load balance router or server to balance the load across the servers...and also possibly mirroring if there are any file writes going on from users.

Why a 700w power supply? Even looking at start-up power draw, 10 drives shouldn't pull more than 150w, plus maybe 100w for the processor, motherboard and RAM. 250w max most likely. You should be fine with a much smaller power supply and that much smaller power supply is likely to be both cheaper and use less power (as you are more in to the sweet spot of efficiency, which is usually around 50% where it is the most efficient). I'd look at a 400-500w 80+ Gold PSU if you are concerned about power efficiency here.

Depending on what is up with everything, I'd look at a back up for that data, either as a warm standby server with periodic backups to it, or else a couple of USB/eSATA enclosures hooked up to the server pulling periodic backups. Thats a lot of data to possibly lose.

As mentioned above, you have to tell use your use case. 30 clients streaming 40TB doesn't really tell us much.

30 full bluray rips at the same time streaming at 1080p is far different than 30 people with 3 simultaneous 12mbps videos. Are they watching the same thing, if so then you might need to multicast which adds network gear. If not, than you have to deal with random read. If all 30 people hit files on 1 of those 4TB drives, even 12mbps files are likely to crush it since random IO floors on disks. +NICS + network gear that can handle the load... etc

I mean our 1500 user Exchange SAN with "only" 20TB of disk was 25k due to IO load to give you an idea of what you might be looking at.

I agree with needing to know more information.

If only a Gbit network than it does not matter about the server as much as that will bottle neck everything enough to effect playback for less than 30 client PC's. Looking to setup load balancing needs multiple network cards with each PC using a different IP for the server or a network switch that can be setup to know how to do load balancing.

Going to 10Gbit networking gear can help, but you still need a pricy hardware switch to take 10Gbit in and split it up to the rest at a slower Gbit speed (assuming not also getting the faster gear for all 30 PC's as well instead of the default Gbit that most boards have now).

With 40TB of data, how it is stored will be a big issue, specifically if power usage is a concern. Spining all the drives just for one user will add up over time with a raid array. Adding more users and you start to need faster drives (good seek times) so going for "green" drives becomes an issue as well (more so if the raid implementation does not like green drives).

Given the large amount of data and the number of clients, do you expect them all to be accessing the same data about the same time or would they all be after something different depending on tastes? If the something different then splitting the load over individual drives can be a better approach as you might only have 3 people on each drive and not have to worry about speed. If everyone wants the same data within a short period of time, then a raid array or even a SSD for new/high usage files might be worth considering.

Another question is are the clients only going to be reading from the media server? no copying off from it? Any writing back to it? The copying is an issue as if one or two copy the data, it will place a higher need on speed as it will effect the ability of others to watch smoothly. Without control of individual speeds, one user can effect the rest (need some sort of software management in this case). With the coping of data to the server, it can be an issue if using a software raid as some have a very slow write time, which will effect people trying to read from the same array at the same time.

Just a point regarding setup costs as 40TB is not a small amount of data (assuming no backup). 10 of the largest most expensive drives can be an issue. But going for smaller more cost effective drives leads to issues with having them connected all at once for use. So even looking at a one off special motherboard, you might have to look at getting some sort of SAS setup for drives or a motherboard with lots of drive controllers.

As one person mentioned before though, the instant power need to power up 10+ drives at once is not a small amount. If you read some of the power specs on some drives, they will report needing 2A on the 12V rail just to start spinning. This drops off when running but still, that is nearly 25W per drive, so 250W for 10 drives and it does not count what the rest of the computer is wanting for that up to a second of time after the power switch is pressed.

Something like this,
Inception 1080p, 2 GB is being watched by 30 clients at the same time,
using samba connection.
Server is already gigE, but switch and client are mb ethernet..
Some people on other forum said it shouldn't be a problem,

1080p movie is 2 GB and 90 minutes long this means an average bitrate of 380 kB/s..
For 30 clients that should be enough ... And that's the worst case, realistically they shouldn't always watch movies the same drive..

But i think i need to replace the switch cause it's still mb, am i correct?

Regarding the power supply, i'm using better safe than sorry approach..
Since i don't want to risk losing data due to power failure...
For 10 HDD accessed at the same time, 700 watt should be enough right?

 

[greenhawk]

Inception 1080p, 2 GB is being watched by 30 clients at the same time,
using samba connection.
Server is already gigE, but switch and client are mb ethernet..
Some people on other forum said it shouldn't be a problem,

1080p movie is 2 GB and 90 minutes long this means an average bitrate of 380 kB/s..
For 30 clients that should be enough ... And that's the worst case, realistically they shouldn't always watch movies the same drive..

But i think i need to replace the switch cause it's still mb, am i correct?

Regarding the power supply, i'm using better safe than sorry approach..
Since i don't want to risk losing data due to power failure...
For 10 HDD accessed at the same time, 700 watt should be enough right?

2GB for a 1080p movie is more of a BluRay Rip than a full movie. The 1080p full copies I have seen are over 30GB per disk. The smaller ones listed on the web are about 10GB each. From there it goes into smaller quality for smaller size (about 5GB for 720p)., So 2GB seems way too small for true 1080p.

When you say the switch and motherboards are only ethernet, do you mean 100Mb/s? If that is the case, then the faster Gbit in the server is going to be running at 100Mb/s as well. about 10MB/s top speed total.

Using truer numbers for required data, 30GB/90Minute, is about 5.5MB/s. A 100Mb/s is 12.5MB/s MAX (generally 9-10MB/s is considered good utilization IIRC given required overheads).

5.5MB/s times 30 users is 165MB/s which is faster than a single Gbit networking connection. If using the 10GB version of the movies (instead of 30GB) , then you need closer to 50MB/s, but that is more than 100Mb/s and less than what a Gbit network can give.

So yes, either way a new switch is needed. Though given the cost of multiple port Gbit units, getting several smaller ones is more cost effective. But that means network layout becomes a big issue as it is possible for the links between switches to be the bottleneck.


Getting back to the drives, a 700W might still be over kill depending on how the power is generated. A large single voltage rail can work just fine. If you get a power supply with separate power rails, then there is the chance to not have access to all the power that the power supply can generate. Though even a good power supply can not protect from issues from the grid.


By the comment of "realistically they shouldn't always watch movies the same drive", I take it to mean that you will not be raiding the drives together to appear as one large shared drive or will you be doing something different? I ask as sorting and managing 10+ drives is not a set and forget process. It takes a lot of time to sort that large amount of data, espically when it expands.

 

[denywinarto]

2GB for a 1080p movie is more of a BluRay Rip than a full movie. The 1080p full copies I have seen are over 30GB per disk. The smaller ones listed on the web are about 10GB each. From there it goes into smaller quality for smaller size (about 5GB for 720p)., So 2GB seems way too small for true 1080p.

When you say the switch and motherboards are only ethernet, do you mean 100Mb/s? If that is the case, then the faster Gbit in the server is going to be running at 100Mb/s as well. about 10MB/s top speed total.

Using truer numbers for required data, 30GB/90Minute, is about 5.5MB/s. A 100Mb/s is 12.5MB/s MAX (generally 9-10MB/s is considered good utilization IIRC given required overheads).

5.5MB/s times 30 users is 165MB/s which is faster than a single Gbit networking connection. If using the 10GB version of the movies (instead of 30GB) , then you need closer to 50MB/s, but that is more than 100Mb/s and less than what a Gbit network can give.

So yes, either way a new switch is needed. Though given the cost of multiple port Gbit units, getting several smaller ones is more cost effective. But that means network layout becomes a big issue as it is possible for the links between switches to be the bottleneck.


Getting back to the drives, a 700W might still be over kill depending on how the power is generated. A large single voltage rail can work just fine. If you get a power supply with separate power rails, then there is the chance to not have access to all the power that the power supply can generate. Though even a good power supply can not protect from issues from the grid.


By the comment of "realistically they shouldn't always watch movies the same drive", I take it to mean that you will not be raiding the drives together to appear as one large shared drive or will you be doing something different? I ask as sorting and managing 10+ drives is not a set and forget process. It takes a lot of time to sort that large amount of data, espically when it expands.

Thanks for the explanation,
yeah i decided to skip the 10+GB rips,
they're just way too large since i have 10,000 movies...
I went for Quantity > Quality approach and use minibrrip 1080p

I'm not sure how good it would be on large tv, (I'm using monitors)
But i think the quality is still acceptable on smaller tvs and monitors..

That, and the fact that larger movies = larger bitrate = lag..

So this should work right?

Server = GB ethernet
Switch = 1 GB port for server, the rest MB ports for client
Clients = all MB port

About the drives, yes i didn't raid it cause i wanted to use the space..
I plan to do so in the future though..
It wasn't anything complicated though, i just setup the samba in ubuntu and map the corresponding networking in the clients...

 

[It's Not Lupus]

Are the server and clients on the same LAN?

If you don't have enough bitrate ("file size") for 1080p, you should use 720p instead.

 

[denywinarto]

Are the server and clients on the same LAN?

If you don't have enough bitrate ("file size") for 1080p, you should use 720p instead.

Yeah like i said before. Im using lan to share movies..
Thats why pre built servers would be overkill
Mini 720p is still blurry sometimes my monitor..
Although not all rips are same
Mini 1080p is a safer bet i think..

 

[smitbret]

Yeah like i said before. Im using lan to share movies..
Thats why pre built servers would be overkill
Mini 720p is still blurry sometimes my monitor..
Although not all rips are same
Mini 1080p is a safer bet i think..

At 2GB, 720p should look better than 1080p, unless it is a static image. If not, then your 720p encodes are crap

 

[denywinarto]

At 2GB, 720p should look better than 1080p, unless it is a static image. If not, then your 720p encodes are crap

Well usually there are 2 options for a movie,
720p and 1080p
720p is usually half the size or 75% of 1080p
that's why i go for 1080p most of the time...

Anything higher than that usually it's in the range of 4GB+ 10+GB
For 4GB i might consider it if the movie has good ratings..

 

[azazel1024]

2GB for a 90min 1080p is probably not going to look very good. Mine are typically in the 4GB range for something around 90min long. Thats more like my typical 720p rip. I tweak my h.264 encode settings a lot to derive maximum quality with minimum size. 2.5Mbps is about the minimum bit rate for 720p I can get and still have a decent video. 1080p its more in the 5Mbps range.

Depends on source though, but that is generally what I find. A 2.8Mbps bit rate 1080p is probably going to look pretty bad, especially in darker scenes (lots of blocking and artifacting). You'd be better off with higher quality 720p rips than much lower quality 1080p rips.

Some of it with the bit rate of the encodes has to do with things like blocking and blur at native resolution versus how well it is upscaled. From what I have found, in general identical bit rate 720p versus 1080p, the 720p generally looks better upscaled to 1080p than the native 1080p looks. That's just for low bit rate encodes though. Once you get to higher bit rates the 1080p looks better than the 720p upscaled for identical bit rates.

You also get in to a lot with how its encoded too. A high profile (or cranked to the max) h.264 5Mbps 1080p is a vastly different beast than a low profile 5Mbps 1080p beast (with the former looking significantly better).

As for the connection, if it is gigabit to a good switch with 10/100 ports and a gigabit uplink port, it might be able to handle it. In the end I think you are better off with gigabit ports on the switch and dual gigabit NICs in the server.

700w power supply is still almost deffinitely overkill. A good 500w is likely to have a decent enough 12v rail to supply all 10 drives with power. They don't all spin up at once.

Using the drives in JBOD is probably your best bet with this. Not sure how Linux does it, but I assume it will only spin up the drives in active use, not the entire JBOD volume. Though I may be wrong about that...if it does spin up the entire volume, I think you are going to want to go with seperate volumes per drive for power/reliability. You can always name each volume seperately. Action/Adventure, Sci Fi, etc to make it a little easier on users looking for a movie. You can always JBOD/storage space together 2-3 drives if a genre takes up more than one drive.

 

[alkemyst]

Sounds like a case of sharing illegally downloaded movies in some kind of communal living arrangement.

Most of the details provided tell us nothing of what bandwidth needs to be supported.