The assessment assumes scepticism regarding possible effect modification. The instrument reflects the generally sceptical view on effect modification found in the theoretical literature and supported by meta-research, including the very small proportion of subgroup explorations that show apparent effect modification. Moreover, attempts to replicate subgroup effects are rare and, if undertaken, rarely successful.1
The assessment is about an association, not a causal relationship. Effect modification refers to an association, not necessarily a causal relationship. A treatment effect may credibly vary among levels of a risk score or body weight, although both are not causes of the effect modification. There might be other causal factors associated with both the apparent effect modifier and the outcome.2, 3, 4, 5, 6 Unless patients were randomised to subgroups defined by the effect modifier, an analysis of effect modification resembles an observational study, even if applied within a randomised controlled trial.2, 5
Magnitude and relevance of effect modification are not part of the assessment. ICEMAN does not directly address the magnitude of effect modification, whether a credible effect modification is important to the patient,7 whether the intervention results in a net benefit when considering multiple outcomes,8 or whether the analysis is appropriate for the research question of interest.9 Importance should be considered independently from credibility and depends on absolute effects, additional outcomes, and context.
Choice of effect measure does not inform credibility. Credibility can be assessed on any scale of interest. There is no general consensus in the methodological literature on how to select the optimal effect measure.10, 11 One approach is, for binary outcomes, to generally prefer relative over absolute scales. Relative effects are more likely to be similar across baseline risk,12, 13 and as a result the heterogeneity of treatment effects is usually substantially lower if one chooses relative rather than absolute effects. Other authors generally prefer absolute effect measures such as risk differences,11, 14 which have some advantages (e.g. calculation of number needed to treat) but also disadvantages.13 A common recommendation is to analyse the data on a relative scale in which true effect modification is unusual, and then, for addressing the magnitude of effect in subgroups when effect modification is credible, calculate magnitude of effects in each subgroup using an absolute scale.15
On using categorical and continuous rather than binary response options: ICEMAN uses four categorical response options for the core items and a continuous scale for the overall assessment divided into four areas. Making the overall assessment continuous instead of categorical results in higher formal ratings of reliability: when two raters differ on a four-point scale, they may in fact almost agree on a continuous scale. ICEMAN’s four credibility areas facilitate reporting and are likely to be useful for consumers of the instrument ratings.
On the decision to offer two separate versions for RCTs and meta-analyses: RCTs are prospective, meta-analyses are retrospective; this has consequences for the relative impact of a priori considerations and the concept of confirmation. Individual participant and aggregate data meta-analysis is not mutually exclusive and combinations of both are possible. Multi-centre RCTs can be conceptually similar to meta-analyses, and in special cases an adapted meta-analysis version of ICEMAN may be helpful to assess effect modification made in multi-centre trials where each centre is treated as a trial.
On the choice of different types of random effects models: Simulation studies have shown that use of a fixed effect model is associated with a higher risk of finding spurious effect modification.16, 17, 18 Recent publications provide preliminary guidance about the choice of model,19 in particular with respect to issues related to meta-analyses of a small number of studies,20, 21, 22, 23, 24, 25, 26 but also state that more research is needed before clear recommendations can be made.
References
1. Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials [Internet]. JAMA Intern. Med. 2017 ;177(4):554–560.Available from:
http://dx.doi.org/10.1001/jamainternmed.2016.9125
2. VanderWeele TJ. On the distinction between interaction and effect modification [Internet]. Epidemiology. 2009 Nov. ;20(6):863–871.Available from:
http://dx.doi.org/10.1097/EDE.0b013e3181ba333c
3. VanderWeele TJ, Knol MJ. Interpretation of subgroup analyses in randomized trials: Heterogeneity versus secondary interventions [Internet]. Ann. Intern. Med. 2011 ;154(10):680–683.Available from:
http://dx.doi.org/10.7326/0003-4819-154-10-201105170-00008
4. VanderWeele T. Explanation in causal inference: Methods for mediation and interaction. 1st ed. New York, NY: Oxford University Press; 2015.
5. Groenwold RHH, Donders ART, Heijden GJMG van der, Hoes AW, Rovers MM. Confounding of subgroup analyses in randomized data [Internet]. Arch. Intern. Med. 2009 ;169(16):1532–1534.Available from:
http://dx.doi.org/10.1001/archinternmed.2009.250
6. VanderWeele TJ, Robins JM. Four types of effect modification: A classification based on directed acyclic graphs: A classification based on directed acyclic graphs [Internet]. Epidemiology. 2007 Sept. ;18(5):561–568.Available from:
http://dx.doi.org/10.1097/EDE.0b013e318127181b
7. Methodology Committee of the Patient-Centered Outcomes Research Institute. Methodological standards and patient-centeredness in comparative effectiveness research: The PCORI perspective. JAMA. 2012 ;307(15):163640.
8. Alper BS, Oettgen P, Kunnamo I, Iorio A, Ansari MT, Murad MH, Meerpohl JJ, Qaseem A, Hultcrantz M, Schünemann HJ, Guyatt G, GRADE Working Group. Defining certainty of net benefit: A
GRADE concept paper [Internet]. BMJ Open. 2019 ;9(6):e027445.Available from:
http://dx.doi.org/10.1136/bmjopen-2018-027445
9. European Medicines Agency. ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. 2020 ;
10. Poole C, Shrier I, VanderWeele TJ. Is the risk difference really a more heterogeneous measure? [Internet]. Epidemiology. 2015 Sept. ;26(5):714–718.Available from:
https://doi.org/10.1097/EDE.0000000000000354
11. Lesko CR, Henderson NC, Varadhan R. Considerations when assessing heterogeneity of treatment effect in patient-centered outcomes research [Internet]. J. Clin. Epidemiol. 2018 Aug. ;10022–31.Available from:
http://dx.doi.org/10.1016/j.jclinepi.2018.04.005
12. Rhodes KM, Turner RM, Higgins JP. Empirical evidence about inconsistency among studies in a pairwise meta-analysis. Res. Synth. Methods. 2016 ;7(4):346–370.
13. Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials [Internet]. Stat. Med. 1998 ;17(17):1923–1942.Available from:
http://dx.doi.org/10.1002/(sici)1097-0258(19980915)17:17<1923::aid-sim874>3.0.co;2-6
14. Lesko CR, Henderson NC, Varadhan R. Considerations when assessing heterogeneity of treatment effect in patient-centered outcomes research [Internet]. J. Clin. Epidemiol. 2018 Aug. ;10022–31.Available from:
http://dx.doi.org/10.1016/j.jclinepi.2018.04.005
15. Varadhan R, Wang S-J. Treatment effect heterogeneity for univariate subgroups in clinical trials: Shrinkage, standardization, or else: Treatment effect heterogeneity for univariate subgroups in clinical trials [Internet]. Biom. J. 2016 Jan. ;58(1):133–153.Available from:
http://dx.doi.org/10.1002/bimj.201400102
16. Higgins JPT, Thompson SG. Controlling the risk of spurious findings from meta-regression [Internet]. Stat. Med. 2004 ;23(11):1663–1682.Available from:
http://dx.doi.org/10.1002/sim.1752
17. Hua H, Burke DL, Crowther MJ, Ensor J, Tudur Smith C, Riley RD. One-stage individual participant data meta-analysis models: Estimation of treatment-covariate interactions must avoid ecological bias by separating out within-trial and across-trial information: One-stage
IPD meta-analysis models must avoid ecological bias [Internet]. Stat. Med. 2017 ;36(5):772–789.Available from:
http://dx.doi.org/10.1002/sim.7171
18. Rubio-Aparicio M, Sánchez-Meca J, López-López JA, Botella J, Marín-Martínez F. Analysis of categorical moderators in mixed-effects meta-analysis: Consequences of using pooled versus separate estimates of the residual between-studies variances [Internet]. Br. J. Math. Stat. Psychol. 2017 Nov. ;70(3):439–456.Available from:
http://dx.doi.org/10.1111/bmsp.12092
19. Veroniki AA, Jackson D, Bender R, Kuss O, Langan D, Higgins JPT, Knapp G, Salanti G. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis [Internet]. Res. Synth. Methods. 2019 Mar. ;10(1):23–43.Available from:
http://dx.doi.org/10.1002/jrsm.1319
20. Borenstein M. Common mistakes in meta-analysis: And how to avoid them. Biostat, Inc.; 2019.
21. Michael H, Thornton S, Xie M, Tian L. Exact inference on the random-effects model for meta-analyses with few studies [Internet]. Biometrics. 2019 June ;75(2):485–493.Available from:
http://dx.doi.org/10.1111/biom.12998
22. Röver C, Knapp G, Friede T. Hartung-knapp-sidik-jonkman approach and its modification for random-effects meta-analysis with few studies [Internet]. BMC Med. Res. Methodol. 2015 ;15(1):99.Available from:
http://dx.doi.org/10.1186/s12874-015-0091-1
23. Bender R, Friede T, Koch A, Kuss O, Schlattmann P, Schwarzer G, Skipka G. Methods for evidence synthesis in the case of very few studies [Internet]. Res. Synth. Methods. 2018 Sept. ;9(3):382–392.Available from:
http://dx.doi.org/10.1002/jrsm.1297
24. Friede T, Röver C, Wandel S, Neuenschwander B. Meta-analysis of two studies in the presence of heterogeneity with applications in rare diseases: Meta-analysis of two studies in the presence of heterogeneity [Internet]. Biom. J. 2017 July ;59(4):658–671.Available from:
http://dx.doi.org/10.1002/bimj.201500236
25. Friede T, Röver C, Wandel S, Neuenschwander B. Meta-analysis of few small studies in orphan diseases: Meta-analysis of few small studies [Internet]. Res. Synth. Methods. 2017 Mar. ;8(1):79–91.Available from:
http://dx.doi.org/10.1002/jrsm.1217
26. Seide SE, Röver C, Friede T. Likelihood-based random-effects meta-analysis with few studies: Empirical and simulation studies [Internet]. BMC Med. Res. Methodol. 2019 ;19(1):16.Available from:
http://dx.doi.org/10.1186/s12874-018-0618-3