Bigger IS better. A bigger phone has more space to put all the fancy hardware in. It doesn't necessarily mean more RAM, a better camera and more sensors, but a lot of times it does. With a smaller phone, it's more for a challenge for the devs to cram all this stuff in there, and more often than not, they will have to sacrifice something that otherwise would be there if not for the small size.
My Note 3 only seemed large at first, but then I realized that it fits in my pocket just fine. Jeans pocket, or cargo pants - not the dress pants you wear to work or wedding. Oh, and if you have a job that requires wearing tight pants, get another job. It sucks when someone tells you what you can or cannot wear, unless it's a uniform, in which case you should be concentrating on your job, not on your fancy smartphone.
The only issue with large phones is that the devs focus on fancy hardware, not in-call quality, so I cant hear shit during calls. : (
Finally, there is a limit to how large a phone should be. I'd say it's right in the neighborhood of 6 inches. Maybe slightly smaller. Any larger and it would be plain stupid. many companies realize that, so they try to expand by other means - smart-watches, smart glasses, wrist wearables, etc, etc. I am looking forward to a future where we will all wear a device on our wrist that can do everything from making calls to constantly monitoring and analyzing vitals, to understanding and translating ALL forms of speech, to projecting holographic images the size of a 60" TV and anything in between.