I have to go with You Only Live Twice because it was the only one in which Sean Connery pretended to be Japanese!
Ironically, my favorite book by far (I read them as a kid) was On Her Majesty's Secret Service, which they wasted by casting George Lazenby as Bond and Telly Savales (!) as Blofeld. I would not mind seeing that remade with Pierce Brosnan, who has pleasantly surprised me.
Favorite scene has to be in Thunderball, where Bond sits playing Solitaire while waiting for a hired goon to come and kill him. He grills the thug at gunpoint for the info he needs, and when the guy tries to shoot Bond, he coolly says, "That's a Smith and Wesson, and you've had your six." and plugs him. Classic understated Connery cool.