LARRY D. YORE and STEPHEN LERMAN
METASYNTHESES OF QUALITATIVE RESEARCH STUDIES IN MATHEMATICS AND SCIENCE EDUCATION Received: 25 March 2008; Accepted: 25 March 2008
INTRODUCTION Current calls for better understanding of available datasets and research results are heard in a variety of political, professional, and academic communities. Much of the momentum behind the USA’s ‘Gold Standard for Educational Research’ (U.S. Department of Education, 2003) is about the need for compelling well-supported generalizations that policymakers, politicians, and bureaucrats can use as foundations for public policy and to make decisions about public education, educational spending, and future directions. Unfortunately, this policy privileges quantitative evidence and appears to disregard high-quality interpretative research evidence. We believe such oversight does not fully recognize education and educational research as a social science that grows both by normal hierarchical development and by the insertion of new theoretical discourses alongside the existing ones. Mathematics and science education have benefited from both approaches to knowledge-building (quantitative and qualitative methods) over the last 30 years. The question is not an either/or issue but one of rigorous and appropriate consideration of both approaches that reflects the research question, development of the problem space, and availability of associated research techniques, procedures, and technologies. Jonathan Osborne (2007), in his remarks as past president of the National Association for Research in Science Teaching, called for more ‘armchair science education’; he claimed that 50 years of research, curriculum development, and implementation have not presented consistent and compelling patterns of outcomes. His quick inspection of three leading science education journals and Google Scholar citations suggested that not enough research synthesis articles have been produced, even when such contributions are highly valued by the science education communities. The call for such syntheses, especially those that used qualitative approaches, applies equally well to mathematics education as it does to science education (Firestone, 1993; Yore, 2003). The Editors of the International Journal of Science and Mathematics Education wish to International Journal of Science and Mathematics Education (2008) 6: 217Y223 # National Science Council, Taiwan (2008)
218
LARRY D. YORE AND STEPHEN LERMAN
encourage rigorous syntheses of the rich collection of qualitative research results (case studies, action research, and other interpretative approaches). METASYNTHESIS Thorne, Jensen, Kearney, Norblit, & Sandelowski (2004) suggested that the pressure for evidence-based health care, which parallels the pressures in education for evidence-based instructional strategies and materials, has promoted scholarly activity distinct from conventional literature reviews. They encouraged and discussed their various perspectives on secondary analyses and other endeavours to deconstruct research studies and construct shared patterns of results across common phenomena. Annells (2005) pointed out that this pressure for generalized knowledge claims and qualitative syntheses in the health sciences has placed qualitative researchers in a quandary. Metasynthesis in medical and health care research appears to be a generic term for a variety of rigorous syntheses of qualitative results across multiple studies or quantitative results that do not afford numerical consolidation, rather than a specific set of procedures that parallels ‘metaanalysis’ of quantitative research results. Thorne et al. believed the resulting “product to be fundamentally different from the original parts, capable of substantiating a more convincing argument about the major theoretical elements with the phenomenon of interest and positioned to advance the science in that particular substantive field more forcefully” (p. 1,343). Finfgeld (2003) identified three types of metasynthesis used in the health sciences: theory building (grounded formal theory and metastudy), theory explication, and descriptive study. She provided specific guidance to form, purpose, and procedures for each of these types—methods (determination of focus, sampling, and data analysis) and integrity of finding (research team, triangulation, supporting evidence, audit trial, and truth value). Fox (2005) suggested that systematic reviews of research and other secondary analyses or metasyntheses could be useful and of increasing interest to government and related fields in deciding critical issues, policy coverage, and intervention effectiveness in the health sciences. We believe that some qualitative synthesis guidelines and procedures developed in and applied to health science research have potential applications to mathematics and science education research. Fortunately, there are a few examples within educational research that illustrate integrated views of qualitative results across a common focus or problem domain. Sadly, some of the most popular and recent books on qualitative research used in mathematics and science education do not mention
METASYNTHESES OF QUALITATIVE RESEARCH
219
qualitative synthesis and only briefly consider the issue of generalizability, if at all, holding to the purist’s interpretation of strict contextual restrictions to qualitative research. This is unfortunate in that much of the impact of high-quality and rigorous interpretative inquiries are having very limited effect on politicians, bureaucrats, and decision-makers since they tend to view each study as an isolated ‘info-bit’ anchored strictly to a unique context or educational setting that cannot be applied widely to their target concerns or constituents. Thus, the strength of qualitative approaches is considered to be an overwhelming weakness. We believe this does not need to be the case. There are several useful approaches for achieving secondary analysis and synthesis of quality research results: systematic research reviews (Bowman, 2007), re-analysis of qualitative data/information with common or similar data sources or anchors, analytical generalizations to a common model or theory (Firestone, 1993), case-to-case generalizations across studies with common focus, data sources, or methods (Rossman, 1993), and other metasyntheses (Finfgeld, 2003; Thorne et al., 2004). SYSTEMATIC RESEARCH REVIEWS Research summaries are wanted and frequently cited by other researchers to capture the background of specific issues in general, to save publication space, and to provide in-depth, comprehensive and readily accessible references to readers. Bowman (2007) pointed out that there are two types of qualitative reviews: non-systematic and systematic. The non-systematic review provides a broad stroke to the background that ‘touches all the bases’, much like the traditional background chapter in theses and dissertations. The International Journal of Science and Mathematics Education is not interested in this type of review. However, “Systematic reviews are a form of research” (Bowman, p. 171) that integrates and synthesizes a selective body of qualitative research requiring thoughtful selections, insightful deliberations, critical analysis, and narrative descriptions to identify the central issues and draw overall conclusions from the primary sources. Systematic reviews ‘add value’ beyond collecting together summaries of prior research in that they critically distil the results from an objective and informed perspective, delineate shared results across researchers or research programs, and suggest additional and further research needed to address voids and emerging hypotheses, models, and speculations. For example, in a study of research texts in two journals and the proceedings of an international
220
LARRY D. YORE AND STEPHEN LERMAN
research group, Lerman, Xu, and Tsatsaroni (2003) were able to show some trends in research methodologies toward qualitative studies in the journals but not the proceedings, changes in focuses of study and theoretical frameworks over a 12-year period, and were also able to draw some conclusions about the relationship between academics and policymakers over that time span.
SECONDARY ANALYSIS
OF
QUALITATIVE DATA
Researchers with access to original information sources generated from a common research focus, agenda, and data collection method across unique settings, informants, or contexts can conduct a re-interpretation of the data using a refined or improved lens or interpretative framework. Hand & McDermott (2008), anticipating the need for such a cross-study synthesis, have re-interpreted the original transcripts from ten independent studies of writing-to-learn science using a consistent interpretative framework for the common anchor interview responses, student-generated response test items, writing samples, and other instructional artefacts. These ‘markers’ allowed the researchers to trace the writing-to-learn science results across several years of their research agenda, to cluster studies for further examination, and to consolidate the information across several small samples. The consolidated results have a higher probability of convincing and persuading stakeholders about the efficacy and effectiveness of this writing-to-learn science approach than the ten independent, isolated studies. Secondary analyses of original qualitative information sources has greater potential with the more common use of discourse analysis software (Atlas ti, Nudist, etc.) and video analysis systems (StudioCode, Transanna, Videograph, etc.) to analyze combined text or video files from clusters of similar studies.
GENERALIZATION
OF
CASE
AND
ACTION RESEARCH STUDIES
Case studies in mathematics and science education have recognized the unique sociocultural, sociocognitive, and contextual features of learning, teaching, and assessment. Therefore, such studies emphasize the uniqueness of the well-defined and bounded context, group, or setting and do not set out to establish generalizations across infinite and undefined situations. Firestone (1993) believed that analytical generalizations and case-to-case transfer could still be achieved with consider-
METASYNTHESES OF QUALITATIVE RESEARCH
221
ation of the theoretical models used and common features across the individual settings. Analytical generalization involves reflections against the theoretical framework, not distinct comparable groups of learners. Here, theory-based or model-driven predictions are made from the theoretical foundations, and these predictions can be supported, confirmed, or rejected, based on the results of individual cases. Firestone stated, “Analytical generalization attempts to show that a theory holds broadly across a wide variety of circumstances, ... that is, the conditions under which it applies” or does not apply (p. 17). Case-to-case transfer simply involves the consideration of independent cases with common focus, method, or outcomes as individual cases in a multi-case study or in a collection of similar cases. This can be achieved from the outset of the research project as an integral part of the design (Florence & Yore, 2004) or across a set of independent case studies judged to have common contextual features, focus, and data sources (Rossman, 1993). Stake (1994) suggested that researchers explore several situations in which a common or similar phenomenon, event, or population occurs and then consider the combined cases as the ‘collective case’. Here, assertions flowing from one case can be deductively verified by information or data from other cases, or alternatively, the analysis can consider the cases as individual data sources. The synthesis across the independent cases is driven by inductive (common assertions inferred across somewhat different cases), interpretive (constructing new assertions from the collection), and analogical (use of compare/contrast strategies to generate distinctions) processes focused on revealing assertions common to the collective case (Rossman). There are many differences between action research and case studies in education, but there is potential for synthesis across similar studies. Action research grew out of the 1970s, a period in which some teachers came to feel disempowered by university researchers coming into their classrooms, observing them, and then going away to write up the research in esoteric journals. Furthermore, those teachers could see no benefits for their pupils or for their own teaching. Today there is a range of perspectives in action research from an emancipation agenda to more immediate and practical purposes—improving performance, understanding practice, and implementing innovations, etc. Action research does not aim for generalizability, but absolute adherence to this basic assumption underutilizes the results. The goal is to make a change in a particular situation; and if readers of the reports detect relevance or significance for other situations and contexts (often referred to as resonance), the application to a different target setting is all
222
LARRY D. YORE AND STEPHEN LERMAN
well and good. From another point of view, most action research intends to describe, in a rich but justified sense, a particular feature of education, be it the effects of curriculum innovation, the state of teachers’ content or pedagogical-content knowledge, or common misconceptions in a particular topic. Action research does not just tell a story about how one teaches or documents the conditions in a classroom, nor should it fall into a strong relativist position where anything goes. Action research is a cyclic process of inquiry and reflection consisting of identifying the problem, planning a change and the data collection related to that change, implementing the change and collecting the data, analyzing the effects of the change, reflecting on the goals and evidence of change, and making a new plan. The data may be qualitative, quantitative, or both. In this way, action research—like case studies—becomes as valid and reliable as other forms of research and candidates for qualitative synthesis described earlier. Closing Remarks The science and mathematics education research communities, although not as long-established as some others such as medical research, psychology, or sciences and mathematics themselves, have nevertheless reached a degree of maturity such that we can speak with authority about what we know about the teaching and learning of science and mathematics and how we have come to know what we know. We can feel confident about presenting our findings from the deep studies enabled by qualitative research as well as from analyses of large datasets. We are perhaps not so adept at disseminating our findings, particularly where the research is qualitative; and we, therefore, do not succeed in reaching the policy-makers and funding groups with knowledge they should have to improve the learning and hence the life chances of today’s school students. We believe that studies of the kinds we have discussed here are crucial in this enterprise; and we encourage researchers to carry out this work, aided we hope by the advice we have offered. The International Journal of Science and Mathematics welcomes the submission of such research. REFERENCES Annells, M. (2005). A qualitative quandary: Alternative representations and metasynthesis [Guest editorial]. Journal of Clinical Nursing, 14, 535–536. Bowman, K.G. (2007). A research synthesis overview. Nursing Science Quarterly, 20, 171–176.
METASYNTHESES OF QUALITATIVE RESEARCH
223
Finfgeld, D.L. (2003). Metasynthesis: The state of the art—so far. Qualitative Health Research, 13, 893–904. Firestone, W.A. (1993). Alternative arguments for generalizing from data as applied to qualitative research. Educational Researcher, 22(4), 16–23. Florence, M.K. & Yore, L.D. (2004). Learning to write like a scientist: Coauthoring as an enculturation task. Journal of Research in Science Teaching, 41, 637–668. Fox, D.M. (2005). Evidence of evidence-based health policy: The politics of systematic reviews in coverage decisions. Health Affairs, 24, 114–122. Hand, B. & McDermott, M. (2008, January). Qualitative synthesis of writing-to-learn science studies. Paper presented at the international meeting of the Association for Science Teaching Education, St. Louis, MO. Lerman, S., Tsatsaroni, A. & Xu, G-R. (2003). Developing theories of mathematics education research: The ESM story. Educational Studies in Mathematics, 51(1–2), 23– 40. Osborne, J. (2007). In praise of armchair science education. E-NARST News, 50(2), 8–11. Rossman, G.B. (1993). Building explanations across case studies: A framework for synthesis. Resources in Education. (ERIC Document Reproduction Service No. ED373115). Stake, R.E. (1994). Case studies. In N.K. Denzen & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 236–247). Thousand Oaks, CA: SAGE Publications. Thorne, S., Jensen, L., Kearney, M.H., Norblit, G. & Sandelowski, M. (2004). Qualitative metasynthesis: Reflections on methodological orientation and ideological agenda. Qualitative Health Research, 14, 1342–1365. U.S. Department of Education (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC: National Center for Education Evaluation and Regional Assistance. Retrieved September 20, 2007, from http://www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf. Yore, L.D. (2003). Quality science and mathematics education research: Considerations of argument, evidence, and generalizability. School Science and Mathematics, 103, 1–7. Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts. Journal of Advanced Nursing, 55, 311–318. Larry D. Yore Curriculum and Instruction (EDCD), MacLaurin Building (MAC) A549 University of Victoria 3800 Finnerty Road, Ring Road, Victoria, V8P 5C2 BC, Canada E-mail: lyore@uvic.ca Stephen Lerman London South Bank University London, UK E-mail: lermans@lsbu.ac.uk