Preliminary considerations
The assessment starts with a set of preliminary considerations to define the apparent effect modification under consideration.
State a single outcome and time-point of interest
Because ICEMAN refers to a single outcome at a time, users must specify the outcome of interest and, if applicable, the time-point of outcome assessment (e.g. mortality at 1 year follow-up).
State a single effect measure of interest
Specify a single effect measure of interest (e.g. relative risk, risk difference, odds ratio, or hazard ratio for binary outcomes, or difference or ratio of means for continuous outcomes). The type of effect measure is a key consideration because the magnitude of effect modification typically differs between effect measures, and in particular between measures of relative versus absolute effect.1, 2, 3, 4 Therefore, the credibility rating is likely to differ depending on the chosen effect measure.
Example: An RCT showed that a lifestyle modification program can prevent diabetes.5 A subgroup analysis divided patients into four groups according to their predicted risk of developing diabetes. On the relative hazard ratio scale, the effect was consistent across risk groups (no suggestion of effect modification). On the absolute risk difference scale, however, the effect was much greater in high-risk than in low-risk patients.6
State a single potential effect modifier of interest
Specify the potential effect modifier of interest (only one effect modifier per ICEMAN form). Effect modifiers may be patient characteristics (e.g. disease severity, age, or type of tumour), intervention alternatives (e.g. different doses, co-interventions, or modes of administration), or, in a meta-analysis, methodological study characteristics (e.g. risk of bias, outcome definition, type of funding). Note that the instrument does not apply when the effect modifier is another outcome. Note that an effect modifier (e.g. sex) is different from a particular subgroup (e.g. women).
ICEMAN applies to effect modifiers assessed before or at randomization (e.g. baseline variables). If the effect modifier is measured after randomization (e.g. an intermediate outcome), the assessment of effect modification is complicated and potentially misleading.7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 Those analyses require different methods13, 17, 22 and result in less secure conclusions.
Exceptions – the instrument does apply to post-randomization effect modifiers if: (1) the effect modifier is a non-modifiable characteristic such as sex or age; or (2) for meta-analyses, the effect modifier is a study characteristic such as risk of bias, length of follow-up, or mean received dose.
Example: An RCT testing strict or conventional management of hyperglycaemia with insulin therapy in ICU patients claimed an effect modification by length of hospital stay.23 Length of ICU stay (the apparent effect modifier), however, was shortened by the intervention. This prognostic imbalance between intervention and control group within the length-of-stay subgroups likely created the differences in mortality.
Does the analysis suggest possible effect modification? (interaction p ≤ 0.1)?
Do not apply ICEMAN if the interaction p-value is larger than 0.1, i.e., provides very little statistical support for the existence of an effect modification. ICEMAN is designed to address the possible claim (presence) of an effect modification rather than the claim of no effect modification (absence).
ICEMAN does not currently provide a version for assessing absence of effect modification / consistency of effects across subgroups.
If the interaction p-value is > 0.1, stop the assessment, report that you did not apply ICEMAN, note the lack of statistical evidence for effect modification. Use the overall effect estimate for drawing conclusions.
Concept and scope of ICEMAN provides additional background information.