Anytime you introduce another USB device to your system, you're introducing a delay. Drivers have to be loaded, data accessed, hardware triggered.. It all takes time. The simpler you keep your system, the faster it is.
You might try disabling 'Legacy USB' in bios, or moving it to a dif USB port, but it's probably driver related. Check the manufacturers site for updated firmware or drivers.