A conversation exploring a key paper that addressed what is (or is not) being learned from RCTs in educational research
Conversation context: The discussion is based upon a paper published in 2019. Matthew Inglis is one of the authors of that paper and he participates in the discussion below. Matthew has also written an outline of the paper, it is published here. (The outline may be a useful short read before listening to the conversation.)
Conversational aim: The central aim is to better understand how the role and contribution of educational RCTs can be fairly assessed. Many commentators have questioned the value of RCT designs in educational research. The work discussed below offers a unique evaluation of how far RCTs actually do achieve what they promise. Matthew’s project analysed results from a very large number of such RCT reports. In brief, findings from this synthesis suggest that RCTs in education rarely are very informative – the effects they find (between intervention and non-intervention groups) are surprisingly small. A particular aim of the conversation is to explore how such a result – surprising and sobering – can be understood.
Conversational themes: What defines a randomised control trial? How is the informativeness of an RCT expressed? What does the present study reveal about the informativeness of a large sample of recent educational RCTs? How might we explain their disappointing outcomes? How should the balance of educational research investment needs to change?
A transcript of this conversation is available. Here as a Microsoft Word file and here as PDF
The questions asked
1.10: “RCTs are the gold standard of educational research” Is that a fair claim of how RCTs stand in the educational community at present?
2.00: In your paper evaluating RCT research – what did you take as a definition of an RCT?
4.50: So in evaluating RCT projects … what can be the research method for evaluating a research method?
7.13: What is an “effect size” in intervention research and how does it differ from the traditional reporting of the outcomes of traditional experiments in terms of “statistical significance”?
11.15: Can you anchor the raw number that is a standardised effect size to some outcome that is more familiar – like a grade difference at GCSE for example?
14.01: But for practitioners, doesn’t there need to be a language of some kind that translates a raw number (effect size) into something more meaningful or familiar?
15.45: Do you accept the point that it is still important to find a way of communicating to practitioners the significance of these kinds of intervention and their effect size numbers?
16.18: Were the trials that were in your sample representative of educational RCTs being conducted at the present time?
18.01: Can you explain how “informativeness” is pinned down by you for this exercise?
21.29: In the end, is it fair to say that your scrutinising of effect sizes reported in your sample of projects revealed very disappointing levels of effect?
21.56: Is it possible that in some large intervention that was not effective overall, there might still be embedded in the results a sub group enjoying high impact effects – say, small groups of individual students or unanticipated features of particular contexts?
26.30: Does your first hypothesis addressing this ‘disappointing’ outcome of evaluating RCTs suggest that interventions need to be grounded in richer theoretical reasoning or deeper understanding of the underpinning learning processes?
29.27: Second hypothesis: What is meant by the notion of “translation” in researchers moving from exploratory/formative research to the specification of a final intervention design?
34.05: Third hypothesis: is the problem simply researchers not being careful in the design and conduct of their projects?
38.02: Of these three possibilities, do you have a favourite for its urgency?
39.48: Are research funders responding to the sobering results reported in this study? [NB: see this link]
45.57: So you are not sympathetic to positions of extreme scepticism about the RCT method in education?
47.12: Is there an inconsistency if a sceptic welcomes RCT research before they select a medical intervention (e.g., vaccination) – while them being intolerant of RCTs for making educational decisions?
48.18: Do people too easily extrapolate from the ease of effective RCT implementation in medicine to the challenge of implementation in the more nuanced contexts of education?
Authored by:
Matthew Inglis
Matthew Inglis is Professor of Mathematical Cognition at Loughborough University
Matthew completed undergraduate (BSc Mathematics) and postgraduate (MSc Mathematics Education, PhD Education) studies at the University of Warwick. Following a period as a Research Fellow at the Learning Sciences Research Institute in Nottingham, he took up a lectureship in Loughborough in 2008.
Charles Crook is Emeritus Professor of Education and a member of the Learning Sciences Research Institute at the University of Nottingham
Leave a Reply