neither SSE or 3dnow can help rc5. its pure integer. even the integer extensinos of SSE and the extended 3dnow instructions don't do anything for it. Don't ask me why, as I could partialy exlpain it, but not well enough to satisfy most people.
If you're talking about S@H, well, the guys at Dcypher were pretty ticked about it, as they developed 3dnow support (or something like that) and decided to create their own distributed computing company. S@H can get it, they chose not to, at least, for now. maybe in another version.