InfiniBand questions

[Jeff7181]

I have a class group project that involves redesigning the network for a bunch of city offices. The offices include two data centers, three main offices and 30 branch offices.

The server and storage solution has been left up to me. One of the key requirements is the ability to expand and the instructor said the more cutting edge technology we implement the better.

So I'm looking at an IBM BladeCenter S and Isilon's clustered storage. The Isilon system is pretty self contained and has two basic interfaces... Gigabit Ethernet and Infiniband. Since Infiniband provides more bandwidth and would be more futureproof I decided it would be best to go with that.

What I'm having trouble with is how to get clients using standard CAT5e and RJ45 connectors physically connected to the Isilon cluster. Would it be as simple as sticking a router in between or would I need other equipment?

It's been difficult to get any kind of response out of IBM, Isilon or Cisco since there's not a change I'll buy any of the equipment I'm asking about.

 

[drebo]

I don't know about infiniband, but with FiberChannel, you need special FiberChannel host bus adapters (HBAs) and FiberChannel switches in between.

I would imagine that Infiniband is probably pretty similar. You'll need to make sure that the IBM BladeCenter S supports an Infiniband module in its blade fabric.

 

[nightowl]

From what I can gather the Infiniband connections are for intra-array communications only. For communication outside of the storage array normal Ethernet is used. See the diagram in the link.
http://www.isilon.com/products/index.php?page=main

IMO, this is great for data storage but if you are looking for truly high performance storage you need to look at Fiber Channel. This is where all of the high performance servers will be connected to the SAN. The Isilon has its place in the network and there are plenty of servers who can take advantage of a GigE connected NAS. However, for your I/O heavy applications Fiber Channel is needed.

Now, if you are looking for something forward looking, talk about multi-fabric I/O. This is what the market is moving towards. It is not something that is completely available today but it will be later in the year. All you need is the right parts and you will be ready to go with the exception of a few connections. Infiniband's big play is for compute clusters where latency is of the utmost importance (talking about ns for latency).

 

[Jeff7181]

Thanks drebo, I checked and it doesn't... looks like I'll have to go with the BladeCenter H.

I also learned a little more about InfiniBand... I think what I need is an InfiniBand module (maybe more than one?) for the BladeCenter, a Cisco SFS 7000 series switch... and then a Cisco 3500 series multifabric switch to connect to my 6500 series switches.

Does that sound about right?

 

[nightowl]

Jeff, take a look at the link I posted. Infiniband is only for the storage cluster back end. You are not going to connect a server via Infiniband to the storage cluster. GigE is all that you are going to be able to connect to the storage cluster at unless you have seen something that I have not.

 

[Jeff7181]

Somehow I missed that diagram, nightowl. That basically fills in a big gap that I was filling with assumptions. I was thinking I'd be able to connect the servers to the Isilon cluster via InfiniBand. Apparently not... that simplifies things quite a bit then. InfiniBand switches for the backend to connect the nodes, and just GigE to connect to the servers. I wanted to be able to have some storage on the Isilon cluster available for clients without a server sitting in the middle... maybe that's not a smart idea though, huh?

 

[nightowl]

With the Isilon storage you were looking you were correct in not having a server between the storage and the server, kind of. There is processing on all of the Isilon's units that run an OS and serve up protocols such as NFS, CIFS, and other application layer protocols. This is also the case with Fiber Channel and like Isilon, the OS is highly optimized for its intended use. However, you can boot directly off a Fiber Channel array like you wanted. You can also boot off of GigE using iSCSI which eliminates the need for an HBA but you do need a NIC that supports iSCSI boot (commonly referred to TOE support).

If you are still looking for new technology take a look at this link for I/O consolidation on servers (The next big trend).

http://www.cisco.com/en/US/pro...per_DC_IO_Consolid.pdf