Hornets are very aggressive. So they very well may attack a person even if you don't think you're bothering them.
We used to camp in a very out of the way area.
I had an uncle used to swat any deer flies etc just hanging out drinking a beer.
He'd lay him on the stump next to him, a Hornet would come down bite the head off the fly and cart it off.
Was like watching a fighter plane come in for a landing if ya didn't make him aggressive.
I guess he must have been a professional Hornet trainer, or been more old school and knew more than I.
*EDIT* and actually yeah before we went down there we would mow and burn out any yellow jacket nests.
Those things in the ground are a lot worse if ya miff em off.