Practical Considerations

Practical guidance for applying, reporting, and interpreting ICEMAN assessments.

Practical considerations

Assessment in duplicate

Confidence in the assessment increases if two investigators independently apply ICEMAN, discuss discrepancies, and present a consensus version.

Reporting

We recommend specifying use of ICEMAN in the study protocol, and in the methods, results, and interpretation sections of the final publication:

  • Protocol: “We will assess the credibility of potentially relevant effect modification using ICEMAN.”
  • Methods: “We used ICEMAN to assess the credibility of potentially relevant effect modification.”
  • Results: “We judged the credibility of the potential effect modification as low, with uncertainty arising from lack of prior evidence and an inconclusive test of interaction (see supplement).”
  • Interpretation: “A formal credibility assessment rated the apparent effect modification as likely spurious. We recommend considering the overall effect estimate for all patients.”
Warning

We do not recommend reporting overall credibility as a percentage (e.g. “30% credible”).

Using ICEMAN with other instruments

ICEMAN can be combined with the Cochrane Risk of Bias tool for RCTs1 or the ROBIS tool for systematic reviews,2 and with the GRADE framework3:

  • Moderate or high credibility: Apply GRADE to subgroup-specific estimates. Note remaining uncertainty if moderate. Considering subgroup-specific estimates may sometimes resolve concerns due to heterogeneity and consequently increase certainty of evidence and strength of recommendation.
  • Low or very low credibility: Apply GRADE to the overall effect estimate. Note remaining uncertainty if low, especially if the potential effect modification appears to explain heterogeneity.

References

1. Higgins J, Sterne J, Savović J, Page M, Hrõbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials. Cochrane Database of Systematic Reviews. 2016 ;1029–31.
2. Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R, ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed [Internet]. J. Clin. Epidemiol. 2016 Jan. ;69225–234.Available from: http://dx.doi.org/10.1016/j.jclinepi.2015.06.005
3. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Glasziou P, Jaeschke R, Akl EA, Norris S, Vist G, Dahm P, Shukla VK, Higgins J, Falck-Ytter Y, Schünemann HJ, GRADE Working Group. GRADE guidelines: 7. Rating the quality of evidence–inconsistency [Internet]. J. Clin. Epidemiol. 2011 Dec. ;64(12):1294–1302.Available from: http://dx.doi.org/10.1016/j.jclinepi.2011.03.017