Actually I thought about it, and it doesn't take doing something to destroy time.
By sending someone back in time, it was done in the future, and for the people in the future it was already done. As a result they have no reason to send back in time, therefore it wasn't done, so they send back in time, therefore there is no reason to send back...
You get the picture, even sending something 1ns forward or backward will instantly cause time to stop. It will just STOP dead, and never exist in the future or the past, or the present.