You're missing my point. You're mentioning the classic "scientific method" model where basically everything is being done by forming a hypothesis and then using prospective experimentation and controls such as double-blind testing to test the hypothesis and verify causality and that it's repeatable. If instead we pretty much exclusively use retrospective tools like data mining and predictive analytics then doesn't that flip the entire foundation of the scientific method as there's no inherent need to establish a hypothesis or causality to get useful results. Indeed, the entire concept of "science" may bifurcate like what happened to neurology and psychology where one primarily concerns itself with the physical brain and nervous system and the other with the more intangible and philosophical "mind" and mental health.
Worse yet are some of the excessive fees to read these journals. If I recall correctly, a lot of these journals have been gobbled up by larger companies. They're making fistfuls of dollars in profits - no one gets paid for the journal articles, and no one gets paid for the peer review. That's a system that needs to be fixed - free market should not apply; they're really just taking advantage of academia.I would add to this that published studies are often behind journals' pay-to-read firewalls, and it's impossible for the lay public to get access for a reasonable fee.
What I would like to see is the major media collaborating on funding a site the provided more comprehensive analyses of important studies, addressed to laymen. That way, a newspaper could still provide its own short write-up and provide a link to much more in-depth articles.
Nothing will change in light of this though is the sad part.
The scientific method has changed. Instead of making an assertion, hypothesizing, testing, and analyzing your data we just take a statistical study that is DEFINITELY not bias for interested parties - then draw some random conclusion that doesn't even correlate with any REAL scientific testing.
It's sad to say, but I don't know when this will stop. It's much like the media - people will do things that get attention rather than do important things.
Worse yet are some of the excessive fees to read these journals. If I recall correctly, a lot of these journals have been gobbled up by larger companies. They're making fistfuls of dollars in profits - no one gets paid for the journal articles, and no one gets paid for the peer review. That's a system that needs to be fixed - free market should not apply; they're really just taking advantage of academia.
there's a lot of people doing studies who are bad at statistics. at least, that's what a clinical statistician tells me.
Without wanting to seem flip, should the scientific method change though? In the era of big data, massive computing power, and current scientific theories that not only defy sound logic but the means to test them (see most of the field of quantum physics, concepts like "dark energy," or even actual observable things like this new space drive), does it still make sense to go through the entire course of the scientific method? Seems kinda pointless to develop a hypothesis and test it when for example one can just consult the entire CDC database and determine which treatment worked best from the millions of case studies on file and just use that.
Well feel free to test your hypothesis experimentally, draw conclusions, refine your hypothesis and publish your theory. If it's repeatable and shows improvement over the current scientific method then the scientific community will gladly embrace it.
Nothing will change in light of this though is the sad part.
The scientific method has changed. Instead of making an assertion, hypothesizing, testing, and analyzing your data we just take a statistical study that is DEFINITELY not bias for interested parties - then draw some random conclusion that doesn't even correlate with any REAL scientific testing.
It's sad to say, but I don't know when this will stop. It's much like the media - people will do things that get attention rather than do important things.
Again, why do you even need a hypothesis? If you had access to millions of (sanitized for privacy) patients' medical records and genomic records, you could provide the proper medical treatment for someone without knowing how it worked or why by essentially "brute forcing" the correct course of action out of the countless permutations previously tried in previous patients. Hell, needing to develop a hypothesis for the causal mechanism for how a particular outcome worked might actually slow down or impede scientific progress in the future since understanding the "why" of something might become the least important part of science.
Again, why do you even need a hypothesis? If you had access to millions of (sanitized for privacy) patients' medical records and genomic records, you could provide the proper medical treatment for someone without knowing how it worked or why by essentially "brute forcing" the correct course of action out of the countless permutations previously tried in previous patients. Hell, needing to develop a hypothesis for the causal mechanism for how a particular outcome worked might actually slow down or impede scientific progress in the future since understanding the "why" of something might become the least important part of science.
Again, why do you even need a hypothesis? If you had access to millions of (sanitized for privacy) patients' medical records and genomic records, you could provide the proper medical treatment for someone without knowing how it worked or why by essentially "brute forcing" the correct course of action out of the countless permutations previously tried in previous patients. Hell, needing to develop a hypothesis for the causal mechanism for how a particular outcome worked might actually slow down or impede scientific progress in the future since understanding the "why" of something might become the least important part of science.
A couple of great hypotheses for testing!
- H1: In the presence of large detailed medical records brute forced statistical analysis may yield quicker novel treatments than standard approaches
- H2: In the presence of large numbers of detailed records statistical analysis may result in finding causal linkages faster than hypothesizing and testing for causal linkages
If it's not obvious it's very difficult to propose an idea about how something in the world may work and it NOT be a hypothesis. The hard part is determining an appropriate test to adequately test the hypothesis.
Study design is hard, but perhaps not the barrier you think. The better your study design, the more $$$ it costs. Huge barrier.
RE: your approach, there are countries with socialized medicine that have been collecting data for years, and they generate hypotheses and test them with cross-sectional analyses and cohort studies (usually retrospective). They are useful, but they are near the bottom of pyramid for validity, because they are subject to all sorts of biases that you can't control for.
If you were to "brute force it", by having a computer generate random hypotheses and test them, you would have very little validity. What we do is we fit our data to a model (e.g. normal distribution of outcome), and test the likelihood that we achieved this result by chance. For rejecting that the data was achieved through random chance (alpha), we generate a p-value. Usually we use a p value of .05 for our primary outcome of a study. This means that, if we find a difference, we consider it statistically significant if there is a less than 5% chance that, if the outcome was random according to our given model, then we would achieve this result.
So...95% chance of the result being real, in laymans terms.
So basically if you do what you suggest, then what happens if you test 1000 random hypotheses? 50 of them will find a result that you believe to be true merely by chance.
Not saying that this approach is useless. There are various ways to adjust your P-value when running multiple corrections. Unfortunately, this means that likely such a difference would have to have to be fairly large to believe a result, and you'd get few hits from doing that.
But, that is not the value to your approach that we see. The value to your approach is for generating hypothesis, that we can then design a study, preferably a randomized controlled trial, to test that hypothesis. And then we should repeat that study. And then we should repeat it again. And then we should perform a systematic review and meta-analysis. And then, maybe, we can be pretty sure we know what we're talking about.
Well it's not really my hypothesis. In a round about way I was trying to show Glenn what he was proposing doesn't actually remove hypothesizing and that he was basically hypothesizing that removing hypothesizing would speed up science.
So basically if you do what you suggest, then what happens if you test 1000 random hypotheses? 50 of them will find a result that you believe to be true merely by chance.
I like science as much as I hate bullshit so I'm always glad when something comes along that brings the proper attention to the two.
No, 50 of them will appear to be true (significant) by chance, since anyone doing studies like this would now how to correct for multiple comparisons.
I don't know why you would assume that people would use faulty statistical methods for studies like this (I know that I said earlier, that the majority of scientist only have a rudimentary understanding of statistics, but correcting for multiple comparisons falls within the realm of rudimentary imho).
Well it's not really my hypothesis. In a round about way I was trying to show Glenn what he was proposing doesn't actually remove hypothesizing and that he was basically hypothesizing that removing hypothesizing would speed up science.