Statistical Fragility in Shoulder Research: Can P Values Be Trusted?
Robert L. Parisien MD1, David P. Trofa MD2, Alexander Ment BA1, Patrick K. Cronin MD3, Emily J. Curry BA4, Josef K. Eichinger MD5, William N. Levine MD2, Paul Tornetta III, MD1 and Xinning Li MD1 1Boston University Medical Center, Boston, MA 2Columbia University Medical Center, New York, NY 3Harvard-Combined Orthopaedic Residency Program, Boston, MA 4Boston University School of Public Health, Boston, MA 5Medical University of South Carolina, Charleston, SC
Background:
The purpose of this study was to examine the statistical stability of published trials in the shoulder literature.
Methods:
Clinical shoulder research studies were analyzed in six major orthopaedic journals from 2006 to 2016. Statistical significance was defined as a P value of less than 0.05. The Fragility Index for each study outcome was determined by the number of event reversals required to change the P value to either greater or less than 0.05. The Fragility Quotient was determined by dividing the Fragility Index by the total population.
Results:
Of the nearly 24,000 studies screened, 3591 met initial search criteria with 198 comparative studies meeting final inclusion criteria. Of the 357 total outcome events, 74 were initially reported as significant (P < 0.05) and 283 as not significant (P > 0.05). The 74 outcomes reported as initially significant demonstrated a Fragility Index of only 3 (IQR 1 to 6) with a Fragility Quotient of 0.039 (IQR 0.016 to 0.087). The 283 outcomes reported as initially not significant were slightly more stable with a Fragility Index of 5 (IQR 3 to 6) with a Fragility Quotient of 0.071 (IQR 0.045 to 0.105). The Fragility Index for the entire study, encompassing all outcome events, was only 4 (IQR 2 to 6) with an associated Fragility Quotient of 0.066 (IQR 0.038 to 0.102).
Conclusion:
We advocate triple reporting of a P value, Fragility Index and Fragility Quotient to provide a more comprehensive understanding of trial stability in the published shoulder literature.
Back to 2019 Posters
