PCI-Express null-modem cable and/or loopback

[cirthix]

Do any of you have information regarding what is essentially connecting two computers via a pci-express cable (assume no signal issues, this "cable" will be ~3" of PCB) without any intermediate chips (such as non-transparent bridges or other pcie switch-like devices).

I would like to explore the idea of directly connecting two computers with a pcie (initially x1) electrical interface with a TX to RX crossover cable and writing a virtual device driver that would run on both systems and appear to the host (linux) as an ethernet interface.

Electrically, there should be no issue. My guess is that there will be some sort of hangup in the link initialization or data/transport layer packet encapsulation.

This is for research purposes. If you don't have a good working knowledge of the lower-level operations of pcie, don't come in here and respond "won't work". I'm looking for highly-technical responses here.

Thanks,
cirthix

 

[sao123]

Flow control would seem to be your greatest challenge...
A good place to start would be the pci express flow control specifications document at EEtimes.

 

[Modelworks]

Something is going to have to keep the clocks in sync and provide some sort of buffering and be able to communicate with the host chip to let it know that the resources are in use. I can't see that being done with just a driver because I am pretty sure that the motherboard chips do not have that low a level of software interface on the user side, you might have to delve into the bridge chips firmware.

Making it appear as ethernet is a whole another topic in itself. Being able to send data bits from one host to another without some sort of interface chip will be a challenge in itself.

Something else you might consider is building a pci express bus on a breadboard and trying to get the interfaces to communicate directly without the need to worry about motherboard issues. Even better would be to simulate the interface in something like Proteus or other spice software.

 

[sao123]

Something is going to have to keep the clocks in sync and provide some sort of buffering and be able to communicate with the host chip to let it know that the resources are in use. I can't see that being done with just a driver because I am pretty sure that the motherboard chips do not have that low a level of software interface on the user side, you might have to delve into the bridge chips firmware.

Making it appear as ethernet is a whole another topic in itself. Being able to send data bits from one host to another without some sort of interface chip will be a challenge in itself.

Something else you might consider is building a pci express bus on a breadboard and trying to get the interfaces to communicate directly without the need to worry about motherboard issues. Even better would be to simulate the interface in something like Proteus or other spice software.

PCI express is a packet based architecture and is completely compatible with ethernet and without drivers... the difficulty arises in memory addressing and routing those packets, even between 2 hosts, without some sort of bridge or packet switching interconnect.

PCIe over cable has been on the market for years, however never before without an intermediary.

 

[cirthix]

Thanks for the replies in this thread, good input.

I suspect that the chips that are used on the pcie-over-cable cards are more than just signal boosters and active equalizers, but since I have some PCB space that is essentially free, I'm going to make a crossover x16 board+cable.

 

[sao123]

Thanks for the replies in this thread, good input.

I suspect that the chips that are used on the pcie-over-cable cards are more than just signal boosters and active equalizers, but since I have some PCB space that is essentially free, I'm going to make a crossover x16 board+cable.

i would advise you to be sure of the pinout you are using, and avoid connecting the 12V power pins 9for fear of frying the connected motherboard). PCIe over cable uses only the 3.3V pins.

 

[helpme]

If you are connecting the PCIe switches of two PCs (TX to RX and RX to TX, clock to clock, reset to reset) you will have two major problems.

1) You can only have 1 root complex in a PCIe network (this is the part of the network that finds and sets up devices) Generally your northbridge is the root complex in a PC - the way to get around this is to use a non transparent bridge on one end of the connection. You can't just connect two PCs together without this, because they will both try to enumerate the bus.

2) You'll need to use one reference clock to run both PCs PCIe networks (or have a switch that supports clock isolation). An additional problem is that most PCs use spread spectrum clocking - which your switches must also be able to support.

The main problem here is that both PCs have their own reference clocks for the PCIe network - without a NTB/Clock isolating switch in-between, they cannot talk to each other because they are running in different clock domains.

I design systems at work (stand alone instruments) that use PCIe for communication, so shoot me any other questions you might have...

Do any of you have information regarding what is essentially connecting two computers via a pci-express cable (assume no signal issues, this "cable" will be ~3" of PCB) without any intermediate chips (such as non-transparent bridges or other pcie switch-like devices).

I would like to explore the idea of directly connecting two computers with a pcie (initially x1) electrical interface with a TX to RX crossover cable and writing a virtual device driver that would run on both systems and appear to the host (linux) as an ethernet interface.

Electrically, there should be no issue. My guess is that there will be some sort of hangup in the link initialization or data/transport layer packet encapsulation.

This is for research purposes. If you don't have a good working knowledge of the lower-level operations of pcie, don't come in here and respond "won't work". I'm looking for highly-technical responses here.

Thanks,
cirthix

 

[helpme]

Wow this is an old post - oops!

 

[signalpuke]

I know this is an old thread, but searching google this was right in line with what I need.

This would be the correct device for directly connecting two PC motherboards?
http://www.onestopsystems.com/pcie_over_cable_z7.php
http://www.onestopsystems.com/documents/HIB35-x4_004.pdf

Basically looking for something short term that can connect the two PCs within close proximity without using ethernet (no PoE req, PCIe has more bandwidth) while allowing future expansion (board with fiber to WAN, storage, etc).

 

[helpme]

It looks like it will work, but you will need to create drivers and software to operate the switch on the card in a non transparent mode.

You might need a card for each PC, and then connect them with a cable - at that point you won't be able to transfer data, you'll have to create something to do that yourself.

 

[imagoon]

I know this is an old thread, but searching google this was right in line with what I need.

This would be the correct device for directly connecting two PC motherboards?
http://www.onestopsystems.com/pcie_over_cable_z7.php
http://www.onestopsystems.com/documents/HIB35-x4_004.pdf

Basically looking for something short term that can connect the two PCs within close proximity without using ethernet (no PoE req, PCIe has more bandwidth) while allowing future expansion (board with fiber to WAN, storage, etc).

2 10GBe cards (1 in each computer) would have more bandwidth by a factor of 2 compared to those cards. Meets all your listed requirements. Cheaper also.

 

[helpme]

4 lanes of Gen 2 PCIe (which is the card linked) is 20 Gb/s vs the 10 Gb/s from one 10 Gb Ethernet link. However, the Ethernet solution will be a LOT easier to setup AND you can use more than one port to get more bandwidth.

10Gbase-T cards (copper) are expensive as well, but still probably cheaper than that one stop card. You could also use optical 10G Ethernet cards instead.

 

[imagoon]

Card listed has a 5Gbps switch.

PCIe 2.0 has a max of 500MB/s per channel. So a 4x card could max out at 16Gbps but the switch on the card can't handle it.

--edit--

Also no reason at all to limit yourself to 10Gbase-T. Active coaxial cables are cheap and you can can get Intel dual port SFP+ cards for sub $300. Cables are around $40 at 5 meters.

 

[helpme]

The switch is a 5 GT x 8 lane switch, not 5 Gb total bandwidth. 4 lanes are dedicated for the cable port, and 4 lanes are dedicated to the card edge connector.

So, by using 4 lanes you will have 2000 MBps available bandwidth.

But yes, using GigE you will be saving yourself a lot of hassle (not to mention that you won't have to turn off the PC just to connect the cables!).

 

[imagoon]

From the spec sheet:

PCIe Switch • 5.0 Gbps 4-Lane PCI Express Gen 2

Pretty sure that PEX8608 isn't a full speed chip since its recommended use case is 1 -> 7 for channel sharing.

 

[helpme]

At work I have used PLX 8648 and 8618 switches (x48 lane and x18 lane) that also carry the same 5.0 Gbps tag line, but definitely work in 4x and 8x bandwidths.

If the 8609 switch was limited to 1x switching bandwidths, then why would PLX allow you to configure it in 4x lane mode? The 8609 also advertises a 4.0 GB/s internal DMA engine for users to offload operations to, which would not be possible to use if the internal switching speed was limited to 5.0 Gbps.

Anyway, I don't work for PLX, so I only have my opinion based on past experiences working with their products. Hopefully our responses have been helpful to Signalpuke in moving along with his project.

 

[greenhawk]

some 10Gbit/s network cards would be the easiest to get going I would think.

Though if after more speed, then having a look at thunderbolt between devices as IIRC macs already can do this with thunderbolt, just have to get it working on the PC side of things.

 

[imagoon]

At work I have used PLX 8648 and 8618 switches (x48 lane and x18 lane) that also carry the same 5.0 Gbps tag line, but definitely work in 4x and 8x bandwidths.

If the 8609 switch was limited to 1x switching bandwidths, then why would PLX allow you to configure it in 4x lane mode? The 8609 also advertises a 4.0 GB/s internal DMA engine for users to offload operations to, which would not be possible to use if the internal switching speed was limited to 5.0 Gbps.

Anyway, I don't work for PLX, so I only have my opinion based on past experiences working with their products. Hopefully our responses have been helpful to Signalpuke in moving along with his project.

No problem, I can't argue much with real life vs the spec sheet. I just found it interesting that have similar 8 lane switches with much higher "switch speed" listings.