Except passengers can actually help the drivers because they can see what the driver sees unlike a phone convo (which also takes a hand away from the driver if they're not using a headset), for instance a talking passenger can know when to shut up (or even call attention to) and let the driver concentrate say if someone cuts them off without signaling.
They've conducted studies on this, and it's the talking on the phone bit specifically (NOT the use of the arm/hand) that most hinders one's ability to drive; even with hands-free, individuals have significantly-poorer reaction times than when not on a phone, and/or when talking with someone else in the car.
Part of it is likely just the way our brain works, and part is because when someone else is in the car with you, they either shut up when they see something dangerous up ahead, and/or draw your attention to hazards that you might not notice.