Most video conferencing software is designed to hit a central server and send the video streams there. Much of this is because 2 client machines behind most firewalls are not able to talk directly to each other but they can both talk to a central server.
I seem to remember that you could use MS netmeeting to do a direct call between two clients which is what you want to do, you had to specify the IP address of the target machine.
Yahoo has turbo webcam mode or something that I think does a direct connection but it probably requires you to first establish a connection to yahoo.
I thought some of the linux apps could do direct ones as well but I don't know which (gnomemeeting maybe?)