I had a Lumia 1520, one of the largest smartphones currently available. I say "had" because I unfortunately dropped the phone and cracked the screen while working on my car . It was a little uncomfortable to use at first, but you get used to it. Obviously a 6" phone wont be as easy to hold in one hand as a smaller phone, but it's still actually manageable if a bit awkward: the glossy finish is too smooth on the back of the phone, and you can't really use your thumb to stretch the span of the display to interact with it, you really need to use your other hand to use it effectively. If you have huge hands (like Shaq size) then a 6" phone seems like a boon and could possibly be a perfect fit. The Verizon flagship Windows Phone, the 5" Lumia Icon, would have been my choice had it been available on AT&T, both for the less glossy finish and for the more manageable size. A 6" display really does add value if you use your smartphone for video, browsing, apps, or reading where even a tiny bit more space can potentially make a big difference. A 6" smartphone might enough to forgo buying or carrying a mini tablet for some people who might otherwise need one. I miss my 1520, it was an excellent phone, but not immune to concrete, unfortunately (my own stupid fault for not getting a case for it).
Before that I had an HTC One (M7) which at 4.7" display size was fairly comfortable to use and close to what I'd say is the ideal size for someone like myself with fairly average sized hands (from what I can tell) for a male, though as others have pointed out in reviews it is a bit tall because of the speakers. The M8 is even larger, but I haven't played with one myself yet.
My current primary phone is the HTC First (the Facebook phone, which I picked up as a cheap replacement at $100... don't even use Facebook on it, funnily enough), which at 4.3" feels like a big step back in terms of size, but again I got used to it and it has the advantage of taking up very little space in my pocket. I carry a Lumia 521 as well which is fairly thick but has only a 4.0" screen and is likewise comfortable to hold and used one-handed.
I think above 6" is where you start leaving "phablet" territory and pretty much have to use two hands in order to use it, and I don't think we'll see many successful smartphones push beyond 6" because of that. Small tablets, on the other hand, are perfectly fine at 7" to ~8" because while they can be pocketed (depending on your pockets, of course ) you don't have the expectation of using them one-handed or holding them up to your face as you would a smartphone. Though I'm sure some users do use them in such a manner.
4" screen size is perfectly adequate for me on OS' like Windows Phone where the tiles and Modern UI ("Metro") apps are cleanly designed and large, though more real estate is usually a good thing. On the other hand, I much prefer having a larger screen and higher resolution when using Android, ideally 4.5" to 4.7", but 5.0 to 5" being acceptable, too, especially for stuff like remote networking (SSH, remote desktop) where fonts can be small and there might be a lot of content on screen. iOS - which I don't use on a daily basis but use often enough - I think would benefit from a similar range of screen sizes without being uncomfortable. Same for Firefox OS, which has an icon grid not unlike iOS or Android.
Anything below 4.3" seems too small on anything but Windows Phone, and above 6" seems extreme. Obviously bezel size, overall device dimensions, the size of your hands, and personal preferences are all factors, but ultimately I think the market will settle on 4.5" to 6" as being the range of smartphone screens, with anything larger falling into tablet territory, and anything smaller being fairly niche or very low end. I don't think there needs to be a "one size fits all" when it comes to device size when a range gives consumers options to suit their needs.
I hope the next generation of "mini" versions of flagship Android phones don't increase above 4.5", though, at least not all of them. I'm considering getting the HTC One Mini 2 soon, though, and hopefully that's the perfect fit for me.