If you are using encryption, wireless connections can use additional CPU power, but the primary factors in the speed data can be transferred on a wireless connection are quality of equipment, distance between the devices, and interference.
For testing purposes, put the media server a few feet away from the wireless router and try transferring that same 100MB file again. This will be your baseline transfer speed for wireless and will give you an idea of the maximum speed capability of your hardware when using wireless. If you get good results when close to the router (30-60 seconds to transfer a 100MB file over wireless is respectable) but not when in the desired location then you may have interference, or are just too far away from the router.
Make sure you don't have any of the common sources of interference in your house like 2.4Ghz cordless phones, high power microwaves in the path of the signal, steel framing, etc.
Also, use NetStumbler on a PC with wireless access (the media server would be fine) to see if there are any other wireless networks in the area that could be interfering. If there are, change your router to use a wireless channel that is not currently being used.
Having said this, wireless CAN be used to stream media once things are set up properly, but due to the many factors that can cause instability in the connection even after it is set up right, going wired really is a better choice for a media server in most cases.