Auto brightness for the most part is nothing but a sham.
I stopped using auto brightness on all my phones when I read this article over 3 years ago.
http://www.displaymate.com/AutoBrightness_Controls_2.htm
So that article shows that autobrightness is all over the place. However, I too have studied autobrightness, and if you look at the CM Jira, I actually specifically opened an issue regarding autobrightness in CyanogenMod and had a lengthy discussion with the developer. The best approximation of how AOSP does brightness with respect to output is a cubic spline.
The article is confusing in that its focused on measuring luminance. Nowhere does Apple or Google say that the slider bar is a linear scale for luminance. Also this site has very limited data. Has anyone bothered to research how newer devices work with auto brightness?
If you read that article, you'll see that "auto brightness" does not in anyway simulate real world conditions.
Once again I think you're focused on the issues of autobrightness rather than what I'm talking about. No where did I say auto brightness simulates real world conditions. I suggested benchmarking with autobrightness because it simulates what most people experience. Benchmarks are about real world benchmarks, no?
Your suggestion won't do much of anything.
If anything, it will simply embolden OEMs like Samsung, LG, and HTC to cheat like they already do today in Android benchmarks.
Hmmm...Testing battery life based on an overly aggressive "auto brightness" settings of a particular OEM and not a fixed luminance like 200 nits? :hmm:
Let the race to the bottom begin...On your mark...get set...GO!!!
The manufacturer's aren't "cheating" on auto brightness right now, and even if they are, that's their loss. You don't think users won't see that their iPhone screen will be dark as hell unless they step out in to the sun or something? The fact is auto brightness works for the vast majority of users, and while it could possibly be better, I, like many other users are perfectly content with it. What makes most users more capable at calibrating a good brightness curve unless we have the right tools? Probably nothing. It's as good as letting users tune RGB to their liking. Just look at the N4 screen calibration thread. Most people are shooting for punchy AMOLED colors and they call that "good."
Until cheating on auto brightness is actually an issue where people feel the need to get OFF auto brightness, it makes more sense to have a benchmark apply to more users than the few users who calibrate their screens to 200 nits, which I'm sure you don't do either.
Just to understand it correctly, you are advocating testing phones exactly as they are delivered by the OEMs with absolutely no modifications done be tech sites to remove cheating and other "questionable" tactics?
There's clear motives in cheating for performance benchmarks where users are obsessing over gigahertz, RAM, and bar charts. Until cheating for battery tests is an actual issue, let's not fabricate a problem.
Do you also believe that reviewers should not bother repackaging GLBenchmark and other Android benchmarks, or create some kind of "StealthBench" and just use the standard benchmarks that have hidden code in TouchWiz to make them run faster on Samsung's devices and never throttle?
I think you're obsessing over the cheating incident with Samsung. These battery rundown tests are typically homemade scripts so it's not something that easy to optimize for. Can you show me where there's battery benchmark cheating right now?
Now if you just take a step backwards from this whole cheating discussion and understand what I'm saying, then perhaps it would make sense. Hypothetically speaking, if the Nexus 5 is too bright in most conditions and eats up 30% more power than other screens, then calibrating at 200 nits would negate that disadvantage. So using a fixed brightness test, one might see the N5 do pretty well, but when they actually use the N5, they find their phone performs significantly worse. Then what? What does that benchmark even mean? How many users will figure out to set their own brightness curves? I guarantee you the most other users (excluding power users) will do is manually set the brightness to some more reasonable level. The users who I see manually set brightness have done it for fear of battery issues, and they typically set their phones far too dim to even be comfortable. So at this point one could argue no battery benchmark matters anymore. I'm simply advocating for benchmarks to show out of the box performance so that people know what they're getting into. If we're all about equalization, then I would push for all phones to be benched on AOSP ROMs and CPUs to be capped at 1ghz.