Exactly like that. Larger values allow for higher throughput of the device that currently owns the bus, but raises the access latency (hence the name) for all the other devices that have a transaction pending and wait for their ownership request to be processed. Set it too low, and transfer rates suffer. Set it too high, and you'll have your realtime streams break up, noticeable as jerky video or scratching, popping sound.
Yes the "right" setting depends on what you have, this is why it's adjustable.
regards, Peter