Advertisement

To the Editor— Misuse of null hypothesis testing: Analysis of biophysical model simulations

Published:December 09, 2016DOI:https://doi.org/10.1016/j.hrthm.2016.12.012
      The recent article by Liberos et al
      • Liberos A.
      • Bueno-Orovio A.
      • Rodrigo M.
      • Ravens U.
      • Hernandez-Romero I.
      • Fernandez-Aviles F.
      • Guillem M.S.
      • Rodriguez B.
      • Climent A.M.
      Balance between sodium and calcium currents underlying chronic atrial fibrillation termination: an in silico intersubject variability study.
      attempts to identify mechanisms that underlie chronic atrial fibrillation termination through the use of a biophysical model of the heart. The manner in which the authors used a combination of model simulations and null hypothesis testing is a concern as we now briefly discuss. The authors generated a population of models by searching parameter space and accepted only those that meet certain criteria. They started with 16,384 possible points in 11-D parameter space, of which 173 met their criteria which were taken forward for further analysis. Therefore, the size of the population, sample size, is controlled by the authors as they control the search space. (Sample size is a key component of a power calculation: probability that a test correctly rejects the null hypothesis when the alternative is true.) This is a concern as the authors then apply statistical tests to their simulated data, without any consideration for power calculations. This concern has been discussed in other fields where they strongly state that applying statistical tests to simulated data as something that should be avoided for multiple reasons, one of which we discuss here.
      • White J.W.
      • Rassweiler A.
      • Samhouri J.F.
      • Stier A.C.
      • White C.
      Ecologists should not use statistical significance tests to interpret simulation model results.
      Since the authors are able to control the sample size in the statistical tests they used to interpret their findings they can control the p-value which renders the null hypothesis test meaningless. An alternative approach to performing null hypothesis testing could have been to report the magnitude of the effect through comparison of the means of the distributions as suggested by White et al
      • White J.W.
      • Rassweiler A.
      • Samhouri J.F.
      • Stier A.C.
      • White C.
      Ecologists should not use statistical significance tests to interpret simulation model results.
      . We encourage the authors to acknowledge that they should not have performed null hypothesis testing and that they consider other approaches to analysing their simulation results. Therefore in this study we would like to encourage the authors to report the means of the distributions of the parameters they considered to be important. This would allow readers to judge for themselves whether the difference in effect is worth pursuing in future experiments.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Heart Rhythm
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Liberos A.
        • Bueno-Orovio A.
        • Rodrigo M.
        • Ravens U.
        • Hernandez-Romero I.
        • Fernandez-Aviles F.
        • Guillem M.S.
        • Rodriguez B.
        • Climent A.M.
        Balance between sodium and calcium currents underlying chronic atrial fibrillation termination: an in silico intersubject variability study.
        Heart Rhythm. 2017; 13: 2358-2365
        • White J.W.
        • Rassweiler A.
        • Samhouri J.F.
        • Stier A.C.
        • White C.
        Ecologists should not use statistical significance tests to interpret simulation model results.
        Oikos. 2014; 123: 385-388

      Linked Article

      • Reply to the Editor—On misuse of null hypothesis testing: Analysis of biophysical model simulations
        Heart RhythmVol. 14Issue 4
        • Preview
          We carefully read the Letter to the Editor regarding our recent publication, in which an experimentally-calibrated population of models was used to investigate differences in response to L-type calcium current block.1 The author speculates on the possibility of controlling the population sample size for yielding statistical significance in the results.2 However, the sample size is primarily determined by the physiologic envelope of the experimental data rather than being manually imposed. Our study goes beyond the use of statistical tests and includes a comprehensive and challenging computational study to evaluate the physiologic significance of the findings, regardless of specific P values.
        • Full-Text
        • PDF