Introduction to Realist Evaluation

What is Realist Evaluation?

Realist Evaluation is a theory-based approach that asks “How or why does this work, for whom and in what circumstances?” rather than just “what works?” The goal is to develop a detailed theory on how a program works and unpack the processes that are normally non-observable.

When to Use Realist Evaluation

  • New initiatives or pilot programs (or programs that seem to work, but for whom and how is not yet understood)
  • Programs for scale-up (to understand how to adapt the intervention to new contexts)
  • Programs that have previously demonstrated mixed patterns of outcomes (to understand how and why this is)

4 Major Steps of Realist Evaluation Framework

  1. Establish Context-Mechanism-Outcome Configuration (CMOC) – i.e. the hypotheses on how a program works
    1. Mechanism: process of how individuals interpret and act upon intervention scheme. Mechanism describes how program brings about effects or change. Often hidden but crucial first step.
    2. Context: conditions into which programs are introduced which are relevant to the program mechanism. Context is important in this framework – need to think about ‘for whom’ and ‘in what circumstances’ a program will work.
    3. Outcome: both the intended and unintended consequences of programs. Outcome patterns are dependent on program context and mechanism, and they can take many forms. Each context into which a program is introduced is unique, which means mechanisms (including reasoning of subjects) activated and outcome patterns will also vary.

CMOC thus are like recipes that describe how different components of the program are harmonised.

EXAMPLE CMO Statement:2. Collect qualitative and quantitative data to investigate the preliminary hypotheses or CMO statements on how the program works. Realist evaluation has no particular preference for either quantitative or qualitative methods, and instead sees merit in multiple methods – marrying the two so that both program processes and impacts can be investigated.

3. Put the CMOC frameworks or hypotheses under a systematic test using data gathered. There is no single analytic method suitable to this test, but ultimately we want to see if this model or a set of hypotheses can explain the complex footprint of outcomes left by the program. When testing the theory, realist evaluators do not rely on a single outcome measure to deliver a pass/fail verdict on a program. Nor do they have a hard and fast distinction between outputs (immediate implementation targets) and outcomes (changes in the targeted behaviour). In fact, program should be tested against a range of output and outcome measures. Sub-group comparisons are needed as outcome patterns of successes and failures will be nuanced based on contexts and mechanisms.

4. Assess and interpret. Have the results of steps 2 and 3 support or refute the theories on how program works? This step is a never-ending or ever-repeating cycle of refining the theory of change/CMOC in order to get closer to better explaining the complex processes behind a black box.

Oltmann, 2011

Summary

Realist evaluation is about theory testing and refinement. Context-mechanism-outcome pattern configurations (CMOCs) comprise models indicating how programmes activate mechanisms amongst whom and in what conditions, to bring about alterations in behavioural or event or state regularities. These propositions bring together mechanism-variation and relevant context-variation to predict and to explain outcome pattern variation. Realist evaluation thus develops and tests CMOCs empirically.

 

Questions? Feedback? Get in touch!
Mona Lee, Evaluation Coordinator; Greater Vancouver Site Coordinator and KTE Coordinator of PLPH; Administrative Coordinator
 
[email protected]