Last Week Tonight with John Oliver: Scientific Studies

[Gonad the Barbarian]

Video: https://www.youtube.com/watch?v=0Rnq1NpHdmw
Ars article on video: http://arstechnica.com/staff/2016/0...ience-journalism-dont-start-with-journalists/

I like science as much as I hate bullshit so I'm always glad when something comes along that brings the proper attention to the two.

 

[TheVrolok]

As always, John Oliver delivers.

 

[s0me0nesmind1]

Nothing will change in light of this though is the sad part.

The scientific method has changed. Instead of making an assertion, hypothesizing, testing, and analyzing your data we just take a statistical study that is DEFINITELY not bias for interested parties - then draw some random conclusion that doesn't even correlate with any REAL scientific testing.

It's sad to say, but I don't know when this will stop. It's much like the media - people will do things that get attention rather than do important things.

 

[Strk]

Nothing will change in light of this though is the sad part.

The scientific method has changed. Instead of making an assertion, hypothesizing, testing, and analyzing your data we just take a statistical study that is DEFINITELY not bias for interested parties - then draw some random conclusion that doesn't even correlate with any REAL scientific testing.

It's sad to say, but I don't know when this will stop. It's much like the media - people will do things that get attention rather than do important things.

The scientific method hasn't changed. People just don't like to acknowledge and/or ignore it. Let's just look at evolution and how a good chunk of this country continues to try and ignore the basic vocabulary of a scientific theory.

 

[Brainonska511]

Nothing will change in light of this though is the sad part.

The scientific method has changed. Instead of making an assertion, hypothesizing, testing, and analyzing your data we just take a statistical study that is DEFINITELY not bias for interested parties - then draw some random conclusion that doesn't even correlate with any REAL scientific testing.

It's sad to say, but I don't know when this will stop. It's much like the media - people will do things that get attention rather than do important things.

As someone in science, I can safely say, the scientific method has not changed. And that is definitely not how it works.

The biggest issue here is the media plays into single-study syndrome. New information must be taken in context with what is already known and conclusions shouldn't be stretched beyond what the data supports (as they often are in click bait headlines). But based on your assertions, the most science you seem to see is directly out of the Today show. Try reading the original works sometime - they don't stretch conclusions beyond what the data supports - it's University PR and the media in general who do that. And it's the public who have been led to believe that science advances in leaps and bounds, when in reality (for the most part), it advances slowly and piecemeal.

As for your conflict of interest mumbo-jumbo - just because someone has a stated conflict of interest doesn't mean the data is automatically bad. There are a lot of great scientists who work in industry and it annoys me, as an academic, to see people besmirched just because they happen to want to also make a paycheck while doing what they love (I don't know if you've looked, but tenure-tracked academic positions are few and far between, and pre-professor salaries are abysmal compared to the private sector). There are companies and individuals guilty of doing bad things, but if anything, most companies want their products to be scientifically valid - they have a vested interest in making sure it really works and not just faking it.

 

[realibrad]

they have a vested interest in making sure it really works and not just faking it.

Airborne

http://www.outsidethebeltway.com/airborne_settles_for_233_million/

 

[Brainonska511]

Airborne

http://www.outsidethebeltway.com/airborne_settles_for_233_million/

Homeopathic garbage is not a drug though. I put those in the snake oil category. Stuff like that and supplements are pushed by people looking to make a buck selling to rubes. It's a shame that crap doesn't fall under more strict regulatory guidelines thanks to Senator Hatch and the relative deregulation that supplements saw in the early 90s.

 

[realibrad]

Homeopathic garbage is not a drug though. I put those in the snake oil category. Stuff like that and supplements are pushed by people looking to make a buck selling to rubes. It's a shame that crap doesn't fall under more strict regulatory guidelines thanks to Senator Hatch and the relative deregulation that supplements saw in the early 90s.

What about Flibanserin - Addyi?

 

[glenn1]

The scientific method hasn't changed. People just don't like to acknowledge and/or ignore it. Let's just look at evolution and how a good chunk of this country continues to try and ignore the basic vocabulary of a scientific theory.

Without wanting to seem flip, should the scientific method change though? In the era of big data, massive computing power, and current scientific theories that not only defy sound logic but the means to test them (see most of the field of quantum physics, concepts like "dark energy," or even actual observable things like this new space drive), does it still make sense to go through the entire course of the scientific method? Seems kinda pointless to develop a hypothesis and test it when for example one can just consult the entire CDC database and determine which treatment worked best from the millions of case studies on file and just use that.

 

[Subyman]

Anyone with a retention of a high school level of scientific understanding bangs their head against a wall every time they hear the media's interpretation of a study. What is funny is that a lot of times the tech advisor for the media outlet would have to have some level of science background to be able to twist the study into saying something else. They are knowingly manufactured for the impact. Others are completely false descriptions, even the abstract usually disagrees with what they are reporting.

 

[hal2kilo]

Homeopathic garbage is not a drug though. I put those in the snake oil category. Stuff like that and supplements are pushed by people looking to make a buck selling to rubes. It's a shame that crap doesn't fall under more strict regulatory guidelines thanks to Senator Hatch and the relative deregulation that supplements saw in the early 90s.

Yep, the herbalist pretty much emasculated the FDA. Snake oil for all!

 

[sandorski]

Without wanting to seem flip, should the scientific method change though? In the era of big data, massive computing power, and current scientific theories that not only defy sound logic but the means to test them (see most of the field of quantum physics, concepts like "dark energy," or even actual observable things like this new space drive), does it still make sense to go through the entire course of the scientific method? Seems kinda pointless to develop a hypothesis and test it when for example one can just consult the entire CDC database and determine which treatment worked best from the millions of case studies on file and just use that.

No, it shouldn't change.

 

[Subyman]

Without wanting to seem flip, should the scientific method change though? In the era of big data, massive computing power, and current scientific theories that not only defy sound logic but the means to test them (see most of the field of quantum physics, concepts like "dark energy," or even actual observable things like this new space drive), does it still make sense to go through the entire course of the scientific method? Seems kinda pointless to develop a hypothesis and test it when for example one can just consult the entire CDC database and determine which treatment worked best from the millions of case studies on file and just use that.

How is your example not using the scientific method? There are plenty of aggregate data studies. They are done all the time. The scientific method is extremely basic. You could break it down into three steps: 1 Come up with a unique idea 2 Test it 3 Draw conclusions from data

If you have a unique idea based on existing data then "testing" has already been done, so you only need to structure the data in a such a way that draws a meaningful conclusion. This doesn't change the method at all.

 

[Meghan54]

How is your example not using the scientific method? There are plenty of aggregate data studies. They are done all the time. The scientific method is extremely basic. You could break it down into three steps: 1 Come up with a unique idea 2 Test it 3 Draw conclusions from data

If you have a unique idea based on existing data then "testing" has already been done, so you only need to structure the data in a such a way that draws a meaningful conclusion. This doesn't change the method at all.

But.....but......but.....Benghazi!

 

[glenn1]

How is your example not using the scientific method? There are plenty of aggregate data studies. They are done all the time. The scientific method is extremely basic. You could break it down into three steps: 1 Come up with a unique idea 2 Test it 3 Draw conclusions from data

If you have a unique idea based on existing data then "testing" has already been done, so you only need to structure the data in a such a way that draws a meaningful conclusion. This doesn't change the method at all.

You're missing my point. You're mentioning the classic "scientific method" model where basically everything is being done by forming a hypothesis and then using prospective experimentation and controls such as double-blind testing to test the hypothesis and verify causality and that it's repeatable. If instead we pretty much exclusively use retrospective tools like data mining and predictive analytics then doesn't that flip the entire foundation of the scientific method as there's no inherent need to establish a hypothesis or causality to get useful results. Indeed, the entire concept of "science" may bifurcate like what happened to neurology and psychology where one primarily concerns itself with the physical brain and nervous system and the other with the more intangible and philosophical "mind" and mental health.

 

[SirStev0]

An attending much smarter than me said the only things that get published are Big Names, Amazing Numbers, or Great Writing.
Facts vs Bullshit play no part in the decision to publish.

 

[woolfe9998]

Great video. The media is indeed terrible in its science reporting. Their bias toward sensationalism tends to cause their reporting to lean in an alarmist direction. Also, they over-simplify research to the point where their version of it barely resembles the research they are reporting on.

They've been doing this with e-cigarettes for several years now. As a result, comparing survey data from now versus five years ago shows that substantially more people now think they are unsafe. But what has actually been occurring in the real world while all this alarmist reporting has been going on is that a mountain of research is showing little if anything to be concerned about with e-cigs.

That is just one example of how the media does a disservice to the public with its science reporting. But it's pretty much all that way. I've gotten to where I won't even finish reading an article on a science topic anymore because it's useless. I just get the names of the researchers out of the article, google them, then go read the study myself. Unfortunately this is necessary if you want to be up to date on science news, because reading an article will give you the wrong impression most times. At best what you are reading is true so far as it goes but highly selective.

 

[realibrad]

Great video. The media is indeed terrible in its science reporting. Their bias toward sensationalism tends to cause their reporting to lean in an alarmist direction. Also, they over-simplify research to the point where their version of it barely resembles the research they are reporting on.

They've been doing this with e-cigarettes for several years now. As a result, comparing survey data from now versus five years ago shows that substantially more people now think they are unsafe. But what has actually been occurring in the real world while all this alarmist reporting has been going on is that a mountain of research is showing little if anything to be concerned about with e-cigs.

That is just one example of how the media does a disservice to the public with its science reporting. But it's pretty much all that way. I've gotten to where I won't even finish reading an article on a science topic anymore because it's useless. I just get the names of the researchers out of the article, google them, then go read the study myself. Unfortunately this is necessary if you want to be up to date on science news, because reading an article will give you the wrong impression most times. At best what you are reading is true so far as it goes but highly selective.

The media is bad, but I dont see why that is the only thing people seem to focus on.

There was also the issue that many of these studies were purposely misleading and done poorly so it could get published and help the researchers keep/get work. The media is a problem, but its also the whole system of publishing research.

 

[shira]

As someone in science, I can safely say, the scientific method has not changed. And that is definitely not how it works.

The biggest issue here is the media plays into single-study syndrome. New information must be taken in context with what is already known and conclusions shouldn't be stretched beyond what the data supports (as they often are in click bait headlines). But based on your assertions, the most science you seem to see is directly out of the Today show. Try reading the original works sometime - they don't stretch conclusions beyond what the data supports - it's University PR and the media in general who do that. And it's the public who have been led to believe that science advances in leaps and bounds, when in reality (for the most part), it advances slowly and piecemeal.

As for your conflict of interest mumbo-jumbo - just because someone has a stated conflict of interest doesn't mean the data is automatically bad. There are a lot of great scientists who work in industry and it annoys me, as an academic, to see people besmirched just because they happen to want to also make a paycheck while doing what they love (I don't know if you've looked, but tenure-tracked academic positions are few and far between, and pre-professor salaries are abysmal compared to the private sector). There are companies and individuals guilty of doing bad things, but if anything, most companies want their products to be scientifically valid - they have a vested interest in making sure it really works and not just faking it.

I totally agree with you about single-study syndrome. But something that does seem to have changed is the increasing misuse of p-values and the growth of p-hacking.

With regard to conflict of interest, the PACE investigation into the effectiveness of various treatment approaches for Chronic Fatigue Syndrome (results released in 2011) seems to have been a classic (and fascinating) case of conflict of interest leading to bad science. Here is a long but great analysis. One of the juiciest criticisms of the entry and exit criteria for that study was:

The study included a bizarre paradox: participants’ baseline scores for the two primary outcomes of physical function and fatigue could qualify them simultaneously as disabled enough to get into the trial but already “recovered” on those indicators–even before any treatment. In fact, 13 percent of the study sample was already “recovered” on one of these two measures at the start of the study.

 

[woolfe9998]

The media is bad, but I dont see why that is the only thing people seem to focus on.

There was also the issue that many of these studies were purposely misleading and done poorly so it could get published and help the researchers keep/get work. The media is a problem, but its also the whole system of publishing research.

Yes, and fault in the one doesn't take away from fault in the other. I was focusing on the media in my post because in the case of e-cigs the media is distorting what the actual research shows. The media is, after all, the only thing that the vast majority of us see - literally the only portal most of us have to the world of science. If there are problems in that realm it's difficult for anyone to even be aware of them because we are not getting an accurate picture media. Worse, the media aids and abets badly done research by reporting it as fact.

 

[shira]

Yes, and fault in the one doesn't take away from fault in the other. I was focusing on the media in my post because in the case of e-cigs the media is distorting what the actual research shows. The media is, after all, the only thing that the vast majority of us see - literally the only portal most of us have to the world of science. If there are problems in that realm it's difficult for anyone to even be aware of them because we are not getting an accurate picture media. Worse, the media aids and abets badly done research by reporting it as fact.

I would add to this that published studies are often behind journals' pay-to-read firewalls, and it's impossible for the lay public to get access for a reasonable fee.

What I would like to see is the major media collaborating on funding a site the provided more comprehensive analyses of important studies, addressed to laymen. That way, a newspaper could still provide its own short write-up and provide a link to much more in-depth articles.

 

[Gonad the Barbarian]

Major media only cares about delivering news, not fact or truth. Sometimes those things cross over, but it's purely coincidental when they do.

 

[Brainonska511]

I would add to this that published studies are often behind journals' pay-to-read firewalls, and it's impossible for the lay public to get access for a reasonable fee.

What I would like to see is the major media collaborating on funding a site the provided more comprehensive analyses of important studies, addressed to laymen. That way, a newspaper could still provide its own short write-up and provide a link to much more in-depth articles.

All federally-funded research has to be made publicly available within 12 months of publication. It may not be instantly readable, depending on the journal, but you can often find recent research deposited in full in places like PubMed (which has had such a policy since 2008). Another good source: the PI's website - they often are allowed to put the full pdf on their website, bypassing the journal paywall.

 

[TeeJay1952]

Ax found, Grinding inevitable.
You can get a "Study" to say what you want.
Examples? Cigarettes, Opioids, Fracking.
Used to be science was involved but nowadays (some)scientists are for sale.

 

[Subyman]

You're missing my point. You're mentioning the classic "scientific method" model where basically everything is being done by forming a hypothesis and then using prospective experimentation and controls such as double-blind testing to test the hypothesis and verify causality and that it's repeatable. If instead we pretty much exclusively use retrospective tools like data mining and predictive analytics then doesn't that flip the entire foundation of the scientific method as there's no inherent need to establish a hypothesis or causality to get useful results. Indeed, the entire concept of "science" may bifurcate like what happened to neurology and psychology where one primarily concerns itself with the physical brain and nervous system and the other with the more intangible and philosophical "mind" and mental health.

I still think you are splitting hairs here. You would still have a hypothesis, you have to or you aren't looking for anything. The "test" would be the organization and analysis of the dataset. Instead of logging ice core samples, for example, you'd simply take the previously recorded data from another source and apply your hypothesis. You would still form a conclusion based on that "test." I'm not sure why you think this is so different.