Publication Bias: The Unseen Factor
Publication bias is an unseen factor that affects the way experiments are published. Generally, we don't publish representative experiments, but instead publish our best experiments. However, when we get different results on different days, it's difficult to decide which are best. This is especially true for mouse experiments, where permission must be obtained to conduct the experiments.
Charles Brenner, PhD
Discovered NR as an NAD precursor vitamin Dept Chair @CityofHope Chief Scientific Advisor @ChromaDex No BS zone Tweets are mine alone
-
there’s something called publication bias that everyone should understand
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
generally, we don’t publish “representative” experiments
we publish our best experiments
how do we decide which are best? -
if we make a mistake in loading a western blot, it’s not very likely to be included in a paper
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
but what about when we get different results on different days?
one set of results favors an exciting idea
the other set are ho-hum -
if we’re honest, the nonreproducibility is spotted in lab mtg and it’s the end of the exciting claims
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
in other labs, multiple ppl are set against each other to get the positive result
ho-hum results are interpreted as “bad experiments” that can be ignored or buried -
positive data are amassed or cherry-picked
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
who would do a thing like this? look at resveratrol
someone claimed that it extends yeast lifespan & activates SIRT1 in 2003
3 papers in 2004-05 showed this to be untrue
2 decades of hand waving
guy still pushing resveratrol in 2023 -
now consider mouse experiments
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
we have to apply for permission to do animal experiments & ensure that our experiments minimize pain & suffering
animal facility staff are required to sac mice that have visible tumors, are in pain or dying -
in practice, lab manager gets an email from animal care saying mouse in particular cage was sac’d
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
so you could start w 2 groups of 25 mice and end up w <50 that finished the experiment -
i’ve yet to see a published partial reprogramming paper state whether their animal facility allows mice to stay in cages w tumors and suffering
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
never seen one that said that they scrapped experiments in which mice got tumors
you need to know this happens. it absolutely happens -
i have seen high profile papers claiming they saw no tumors
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
translation: their protocol didn’t demand that they investigate cause of death (roughly half of lab mice will get tumors)
look at fig 1c in https://t.co/gedJiKSRKo -
does anyone believe they started the experiment w 21 mice in the control group and um…. 15 mice in the partial reprogramming group?
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
also, look at the rate of reported death in the partial reprogramming group
not impressive
and realize i am picking on the field’s best paper -
a 2019 popular book described a faculty member’s conversation w a grad student in 2016 in which they speculated that if they leave out the Myc gene, they could reprogram wo teratomas
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
it’s known that OSK reprogrammed cells do form teratomas in micehttps://t.co/HDjYHdjeD1 -
the public statement (pages 166-169 of lifespan) by a known cherry picker constitutes a gigantic COI
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
he states in a book that OSK is not teratogenic and then publishes the next yr that he safely restored vision w OSKhttps://t.co/Sm7KpdApLq -
the paper states “the investigators were blinded to animal allocation but not cell cultures or outcome assessment”
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
did your jaw drop?
1) guy already claimed counterfactually that OSK reprogramming is not teratogenic
2) animal facility is likely to sac animals w visible tumors -
3) even then the senior author doesn’t insist that the experiment has to be scored in a blinded manner
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
this is the only way to test a hypothesis
if you aren’t blinded and you’ve already announced to the world you restored vision w OSK, you are conflicted -
are you really going to publish mice that got tumors or whose vision got wonky? or are you more likely to collect positive cases and throw out experiments you don’t like?
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
why am i bringing this up now? -
on critique of his cell paper, same individual says no he didn’t reverse aging in cell but he did in naturepic.twitter.com/VDVs0RcL9C
— Charles Brenner, PhD (@CharlesMBrenner) March 18, 2023
so it’s really a house of cards
we know OSK is teratogenic
we know senior author cherry picks data
he’s doing bad science