On similar note, any reason not to use the root servers directly? That's what I do. I have my local DNS server setup to just forward to the root servers.
Internet politeness. The root servers exist purely by necessity, some server has to be the root of each zone. They're not intended to be queried by end users, and all end users querying root servers would lead to global DNS failure. No root server wants you to use it over a caching server. I'm ignoring your private caching server, because everyone having their private caching server would cause the same result.
Set yourself up with DNSSEC if you don't trust the DNS servers you use, and if you're really cautious, configure your local server to not resolve domain names that don't support DNSSEC.
If ISP servers are unreliable, untrustworthy or politically inconvenient, you use something else. Like Google, OpenDNS,
Xiala or any other provider of public DNS servers that suits your needs.