IOSR Journal of Agriculture and Veterinary Science (IOSR-JAVS) e-ISSN: 2319-2380, p-ISSN: 2319-2372. Volume 5, Issue 2 (Sep. - Oct. 2013), PP 51-58 www.iosrjournals.org
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study P.LavanyaKumari Assistant Professor, Dept. of Statistics & Mathematics, Agricultural College, Bapatla, AP, INDIA.
Abstract: Experimental research is important to society - it helps us to improve our everyday lives.Typically, an experiment is constructed to be able to explain some kind of causation. This article is focused onvarious issues involved in comparing related groups and measuring change with pretest and posttest data. Different pretest-posttest designs are discussed in a manner that can help researchers to select suitable test in order to maintain internal and external validity simultaneously. One real time example from agricultural sector is given for better understanding of theme of the paper. Despite its complexity in analysis, Solomon four group methods help the researchers in taking valid conclusions in any field of Research. Findings enumerated for each methodcertainly provide guidance in drawing conclusions appropriately.
Keywords: Solomon four group test, pretest-posttest designs, internal &external validity I.
Experimental designs are often touted as the most "rigorous" of all research designs or, as the "gold standard" against which all other designs are judged. True experimental design is regarded as the most accurate form of experimental research, in that it tries to prove or disprove a hypothesis mathematically, with statistical analysis.In a true experimental study, a control and an experimental group are used. The dependent variable in each group is observed before introduction of the independent variable. Then only the experimental group is introduced to the independent variable. Observations of the dependent variable are made for both groups after the exposure of the experimental to the independent variable. Pretest-posttest designs are an expansion of the posttest only design with nonequivalent groups, one of the simplest methods of testing the effectiveness of a treatment. In this design, which uses two groups, one group is given the treatment and the results are gathered at the end. The control group receives no treatment, over the same period of time, but undergoes exactly the same tests. Statistical analysis can then determine if the intervention had a significant effect or not. One common example of this is in agriculture; one set of plots is given a fertilizer, whereas the control group is given none, and this allows the researchers to determine if the fertilizer really works. This type of design, whilst commonly using two groups, can be slightly more complex. For example, if different levels of a fertilizer are tested, the design can be based around multiple groups. Whilst this posttest only design does find many uses, it is limited in scope and contains many threats to validity. It is very poor at guarding against assignment bias, because the researcher knows nothing about the individual differences within the control group and how they may have affected the outcome. Even with randomization of the initial groups, this failure to address assignment bias means that the statistical power is weak.The results of such a study will always be limited in scope and, resources permitting; most researchers use a more robust design, of which pretest-posttest designs are one. The posttest only design with non-equivalent groups is usually reserved for experiments performed after the fact, such as anAgricultural researcher wishing to observe the effect of a fertilizer that has already been administered. For many true experimental designs, pretest-posttest designs are the preferred method to compare participant groups and measure the degree of change occurring as a result of treatments or interventions.Pretestposttest designs grew from the simpler posttest designs, and address some of the issues arising with assignment bias and the allocation of participants to groups. One example is Agriculture, where researchers want to monitor the effect of insecticides upon yields of wheat. Other areas include evaluating the effects of counseling, testing medical treatments, and measuring psychological constructs. The only stipulation is that the subjects must be randomly assigned to groups, in a true experimental design, to properly isolate and nullify any nuisance or confounding variables.
51 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study II.Terminology 2.1 Confounding Variable In any experiment there are many kinds of variables that will affect the experiment. The independent variable is the manipulation for the experiment and the dependent variable is the measurement taken from that experiment. Confounding variables are things in which have an effect on the dependent variable, but were taken into account in the experimental design. For example, one wants to know if fertilizer X has an effect on getting more yields. The experimenter must take care to design the experiment so that he can be very sure that the yields under observation must be gained because of the influence of Fertilizer X, and that the improvement was not caused by other factors, called confounding variables. 2.2 Internal and External validity External and internal validity are not all-or-none, black-and-white, present-or-absent dimensions of an experimental design. Validity varies along a continuum from low to high. 2.2.1. Internal validity is a property of scientific studies which reflects the extent to which a causal conclusion based on a study is warranted. Such warrant is constituted by the extent to which a study minimizes bias. In other words, to the degree that we are successful in eliminating confounding variables within the study itself is referred to as internal validity. In other words, Internal validity is the degree to which the experimental treatment makes a difference in (or causes change in) the specific experimental settings . The factors that threaten internal validity are: history, maturation, pretest effects, instruments, and statistical regression toward the meanand differential selection of participants, mortality, and interactions of factors (e.g., selection and maturation). 2.2.2 External validity is the validity of generalized (causal) inferences in scientific studies, usually based on experiments as experimental validity. In other words, it is the extent to which the results of a study can be generalized to other situations and to other people. Hence, a study that readily allows its findings to generalize to the population at large has high external validity .Hence, External validity is the degree to which the treatment effect can be generalized across populations, treatment variables, and measurement instruments. Threats to external validity include: interaction effects of selection biases and treatment, reactive interaction effect of pretesting, reactive effect of experimental procedures, and multiple-treatment interference .
III. Types of Experimental Designs 3.1 The Posttest Only Design with Non-Equivalent Control Groups 3.1.1 Requirements of the design It must have two groups. It uses a post-only measure. It has two distributions (measures), each with an average and variation. Assessment of treatment effect = statistical (i.e., non-chance) difference between the groups. This is a two group design including an experimental group and a control group, where assignment to groups occurs through randomization. The experimental group is exposed to the manipulation, the control group is not. There is no pre measurement - a measure of how conditions would have been if there was no manipulation. Both measures are taken after the manipulation occurred. 3.1.2Limitations of Posttest Only Design There is no benchmark. There is no random assortment to treatment groups. This design recognizes the threats associated with pretesting effect and pretest-manipulation interaction bias. www.iosrjournals.org
52 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study 3.2 The Two Group Control Group Design This is, by far, the simplest and most common of the pretest-posttest designs, and is a useful way of ensuring that an experiment has a strong level of internal validity. The principle behind this design is relatively simple, and involves randomly assigning subjects between two groups, a test group and a control. Both groups are pre-tested, and both are post-tested, the ultimate difference being that one group was administered the treatment. This test allows a number of distinct analyses, giving researchers the tools to filter out experimental noise and confounding variables. The internal validity of this design is strong, because the pretest ensures that the groups are equivalent. The various analyses that can be performed upon a two-group control group pretestposttest designs are (Fig 1).
3.2.1 Analysis and interpretation By comparing the final posttest results between the two groups with the help of independent sample ttest, one can find the effectiveness of treatment. (C) With the help of paired t-test one can observe how both groups changed from pretest to posttest, whether one, both or neither improved over time. If the control group also showed a significant improvement, then the researcher must attempt to uncover the reasons behind this. (A and A1) Prior to the above tests, the researchers must compare the scores in the two pretest groups, to ensure that the randomization process was effective using independent sample t-test. (B) These tests evaluate the efficiency of the randomization process and also determine whether the group given the treatment showed a significant difference. 3.2.2 Limitations of Pretest-Posttest Designs The main problem with this design is that it improves internal validity but sacrifices external validity to do so. There is no way of judging whether the process of pre-testing actually influenced the results because there is no baseline measurement against groups that remained completely untreated. The other major problem, which afflicts many sociological and educational research programs, is that it is impossible and unethical to isolate all of the participants completely. 3.2.3 Need for Solomon four group method The two-group control group design is an exceptionally useful research method, as long as its limitations are fully understood. For extensive and particularly important research, many researchers use the Solomon four group methods, a design that is more costly, but avoids many weaknesses of the simple pretest-posttest designs. 3.3 The Solomon Four Group Design This design contains two extra control groups, which serve to reduce the influence of confounding variables and allow the Solomon four group method researchers to test whether the pretest itself has an effect on the subjects.Whilst much more complex to set up and analyze, this design type combats many of the internal validity issues that can plague research. It allows the researcher to exert complete control over the variables and allows the researcher to check that the pretest did not influence the results. The Solomon four group test is a standard pretest-posttest two-group design and the posttest only controls design. The various combinations of tested and untested groups with treatment and control groups allow the researcher to ensure that confounding variables and extraneous factors have not influenced the results. www.iosrjournals.org
53 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study In the figure, A, A1, B and C are exactly the same as in the standard two group design.
3.3.1 Analysis and interpretation The first two groups of the Solomon four group designs are designed and interpreted in exactly the same way as in the pretest-post-test design discussed in previous section, and provide the same tests upon randomization. The comparison between the posttest results of groups C and D, marked by line 'D', allowsus to determine if the actual act of pretesting influenced the results. If the difference between the posttest results of Groups C and D is different from the Groups A and B difference, then the researcher can assume that the pretest has had some effect upon the results. The comparison between the Group B pretest and the Group D posttest allows the researcher to establish if any external factors have caused a temporal distortion which was not included in present study. For example, it shows if anything else could have caused the results shown and is a check upon causality. The Comparison between Group A posttest and the Group C posttest helps the researcher to determine the effect that the pretest has had upon the treatment. If the posttest results for these two groups differ, then the pretest has had some effect upon the treatment and the experiment is flawed. The comparison between the Group B posttest and the Group D posttest shows whether the pretest itself has affected behavior, independently of the treatment. If the results are significantly different, then the act of pretesting has influenced the overall results and is in need of refinement. 3.3.2 Limitations of Solomon Four Group Design The Solomon four group designs is one of the benchmarks for sociological and educational research, and combats most of the internal and external validity issues apparent in lesser designs. Despite the statistical power and results that are easy to generalize, this design does suffer from one major drawback that prevents it from becoming a common method of research: the complexity. A researcher using a Solomon four group design must have the resources and time to use four research groups, not always possible in tightly funded research departments. Most schools and organizations are not going to allow researchers to assign four groups randomly because it will disrupt their normal practice. Thus, a non-random assignment of groups is essential and this undermines the strength of the design. Secondly, the statistics involved is extremely complex, even in the age of computers and statistical programs. Unless the research is critical or funded by a large budget and extensive team of researchers, most experiments are of the simpler pretest-posttest research designs. As long as the researcher is fully aware of the issues withexternal validity andgeneralization, they are sufficiently robust and a Solomon four group design is not needed. 3.3.3 Real time example For the sake of clear understanding, a real time problem has been considered and is solved with three different designs and results are compared.
IV. Methodology 4.1 Research approach Survey method has been adopted to study the key objective of the Research i.e. whether training programme for farmers conducted by Govt.Officials on modern cultivation methods is effective or not. www.iosrjournals.org
54 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study 4.2 Population The population for the present study consists of large farmers who had more than 10 acres of cultivable land in Chittoor district of Andhra Pradesh. 4.3 Sample and Sampling Technique Three Stage Random sampling technique has been adopted to select the sample of farmers. For this study four mandals have been selected and from each mandal one village has been selected randomly. From each selected village, 24 farmers are again selected randomly. Hence, totally 96 farmers in four groups of size 24 haven been selected at random. 4.4 Study variable is Awareness score of farmers on modern cultivation. A tool has been prepared to assess the knowledge levels of farmers with regard to modern cultivation methods. Tool consists of 50 questions and measured with “Yes” or “No” type options. Reliability is found as 8.65 which is good and validity is checked. 4.5 Data A study is designed to assess the awareness of farmers on modern cultivation techniques in agriculture. As discussed in methodology, four groups of farmers had been considered and groups are labeled 1, 2, 3 and 4 respectively. Out of four groups, Group-2 and Group-4 have not been invited for awareness programme on modern cultivation methods conducted by Govt. Officials. Group-1 and Group-3 had attended an awareness programme. Group-1 and Group-2 had been interviewed before and after training whereas Group-3 and Group-4 had been questioned only after the training. Awareness scores are noted in table-1. Notations: Ex1Pre: Awareness Scores of group-1before attending training. Ex1Post: Awareness Scores of group-1after attending training. CG1Pre: Awareness Scores of group-2 before training who are not invited for training. CG1Post: Awareness Scores of group-2 after training who are not invited/attended. Ex2Post: Awareness Scores of group-3 after attending training. CG2Post: Awareness Scores of group-4 after training who are not invited/attended. Table-1: Awareness score of the farmers Group-1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Mean
29 25 30 27 29 29 27 28 26 29 26 27 29 28 30 26 30 27 30 30 27 30 25 27 27.96
32 30 34 34 31 32 35 32 34 30 32 35 32 30 32 31 34 33 32 30 32 30 31 35 32.21
26 26 25 29 29 29 29 25 26 25 30 27 28 26 25 26 28 30 30 27 25 26 29 29 27.29
27 27 29 27 27 29 28 30 27 27 30 29 27 30 26 26 28 26 27 27 28 30 28 26 27.75
35 35 32 31 32 31 34 35 33 35 34 35 35 32 35 35 34 35 34 33 33 34 33 34 33.71
30 25 28 27 25 30 26 29 27 28 25 30 29 26 25 25 25 30 27 29 29 27 27 25 27.25
55 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study 4.6 Statistical Analysis Collected data is properly entered in computer for the sake of analysis. Appropriate Statistical techniques such as Independent sample t-test and Paired t-tests have been applied with the help of SPSS version 20 and results are concluded as per the level of significance. Significant findings are presented graphically.
Results and Discussions
Three different designs had been applied in order to test the main objective and findings have been compared with respect to their effectiveness. 5.1
Solution with the Posttest Only Design with Non-Equivalent Control Groups Line/Fig
Group CG2POST EX2POST
Mean 27.25 33.71
Sd 1.89 1.33
Sig @ 1%
Findings of design 3.1 Using independent sample t-test, it can be concluded that there is significant difference between CG2POSTand EX2POST (without pretest). Since the difference is statistically significant at 1% level, hence it can be concluded that the training is highly effective in enhancing the awareness levels of farmers on modern cultivation methods in agriculture. As a remark, at this point one cannot give assurance whether the existing change only due to training or any other intervening variable which is not included or studied at this time of survey. Hence further refinement over conclusions is required which leads to the following design. 5.2
Solution with the Two Group Control Group Design Line
Group Mean EX1PRE 27.96 A /Fig(1) 1(Pt-test) EX1POST 32.21 CG1PRE 27.29 A1/Fig(1) 2(Pt-test) CG1POST 27.75 CG1PRE 27.29 B/Fig(1) 3(It-test) EX1PRE 27.96 CG1POST 27.75 C/Fig(1) 4(It-test) EX1POST 32.21 *significant @ 5% level ** significant @ 1% level It-test : Independent sample t-test Pt-test : Paired/related sample t-test
Sd 1.68 1.69 1.83 1.36 1.83 1.68 1.36 1.69
Sig @ 1%
Sig @ 1%
Findings of design 3.2 Test#1 suggests that no difference found between before and after values in Control group but Test#2 reveals that there is significant difference between pre and post score of Experiment group which means significant impact of treatment has been found on experiment group. Further Test#3 suggests that the randomization process was effectively applied to get samples under CG1Pre and EX1Pre because no significant difference has been found. Test C ensures that there is significant difference between CG1Post and EX1Post. Hence, it can be concluded that there treatment is very effective in enhancing the scores and perhaps there may be some sort of influence of pre-test on score because the mean difference has been increased from 4.25 to 4.46 from test#1 to test#4.
56 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study 5.3 Solution with the Solomon four Group Control Group Design Line
Group EX1PRE EX1POST CG1PRE CG1POST CG1PRE EX1PRE CG1POST EX1POST CG2POST EX2POST CG1PRE CG2POST EX1POST EX2POST CG1POST CG2POST
Mean 27.96 32.21 27.29 27.75 27.29 27.96 27.75 32.21 27.25 33.71 27.29 27.25 32.21 33.71 27.75 27.25
Sd 1.68 1.69 1.83 1.36 1.83 1.68 1.36 1.69 1.89 1.33 1.83 1.89 1.69 1.33 1.36 1.89
Sig @ 1%
Sig @ 1%
Sig @ 1%
Sig @ 1%
Findings of design 3.3 Test#2 suggests that no difference found between before and after values in Control group but Test#1 reveals that there is significant difference between pre and post score of Experiment group which means significant impact of training has been found on experiment group. Further Test#3 concludes that the randomization process was effectively applied to get samples under CG1Pre and EX1Pre because no significant difference has been found. Test#4 ensures that there is significant difference between CG1Post and EX1Post. Hence, it can be concluded that the training is very effective in enhancing the awareness scores and perhaps there may be some sort of influence of pre-test procedure on awareness scores because the mean difference has been increased from 4.25 to 4.46 from Test#1 to Test#4. Test#5 proves that there is significant difference between CG2POST and EX2POST (without pretest)at 1% level, hence it can be inferred that the training is highly effective. Both Test#4 and Test#5 suggest that there is significant impact of training but mean difference of Test#5 is significantly higher than that of Test#4which is only due to pre-test procedure adopted for the farmers in this study. That means farmers who had not been attended training might be enquired about modern cultivation techniques and enriched themselves with some sort of informationdue to pretest. Interestingly, Test#6 suggests that no external factors have been included unknowingly in this study. Further Test#7 reveals that pretest has had effect upon the awareness scores hence the experiment is flawed. Finally, with the help of Test#8 one can conclude that the procedure of pre-test itself has affected behavior or not. In this example it is proved because farmers who undergone pretest have inquired about modern techniques and might have got some sort of information. All observations from four groups regarding mean awareness scores of farmers have been showed in Fig-3.
Fig-3: Average awareness Score of farmers 40 35 30 25 20 15 10 5 0
57 | Page
Significance of Solomon four group pretest-posttest method in True Experimental Research- A Study VI. Conclusion Experiments are conducted to be able to predict the phenomenon. Present scenario of the research is mainly based on experimental research which provides causality for the variation. There are several designs to find causal relationship among variables. Amongst, Solomon four-group design(Design-3.3) is superior to Posttest only design(Design-3.1) and Two group control group designs(Design-3.2) because, along with controlling for effects of history, maturation, and pretesting, it allows for evaluation of the magnitudes of such effects and higher degree of external validity despite difficulty in conducting the design.
Acknowledgements Author is very grateful to the farmers, and farm workers who assisted wholeheartedly in collecting the data from them at Nellore district.
J. Bellini and P. Rumrill, Research in rehabilitation counseling, Springfield, IL: Charles C.Thomas.
R.D. Bock, Basic issues in the measurement of change. in: Advances in Psychological and Educational Measurement, D.N.M. DeGruijter and L.J.Th.Van der Kamp, eds, JohnWiley&Sons, NY, 1976, pp. 75–96.
A.D. Bryk and H. I. Weisberg, Use of the nonequivalent control group design when subjects are growing, PsychologicalBulletin85 (1977), 950–962.
I.S. Cahen and R.L. Linn, Regions of significant criterion difference in aptitude- treatment interaction research, AmericanEducational Research Journal 8 (1971), 521–530.
L.J. Cronbach and L. Furby, How should we measure change - or should we? PsychologicalBulletin74 (1970), 68–80.
D.M. Dimitrov, S. McGee and B. Howard, Changes in students science ability produced By multimedia learning environments: Application of the Linear Logistic Model for Change, School Science and Mathematics 102(1) (2002), 15–22.
G.H. Fischer, Some probabilistic models for measuring change, in: Advances in Psychological and Educational Measurement,D.N.M. DeGruijter and L.J.Th. Van der Kamp, eds,John Wiley & Sons, NY, 1976, pp. 97–110.
G.H. Fischer and E. Ponocny-Seliger, Structural Rasch modeling, Handbook of the usage of LPCM-WIN 1.0, Progamma,Netherlands, 1998.
R.K. Hambleton, H. Swaminathan and H. J. Rogers, Fundamentals of Item Response Theory, Sage, Newbury Park, CA,1991.
S.W. Huck and R.A. McLean, Using a repeated measures ANOVA to analyze data from A pretest-posttest design: A potentially confusing task, Psychological Bulletin 82 (1975),511–518.
S. Isaac and W.B. Michael, Handbook in research and evaluation 2nd. ed., EdITS, San Diego, CA, 1981.
E. Jennings, Models for pretest-posttest data: repeated measures ANOVA revisited, Journal of Educational Statistics 13 (1988), 273–280.
K.G. J¨oreskog and D.S¨orbom, Statistical models and methods for test-retest situations, in: Advances in Psychological andEducational Measurement, D.N.M. DeGruijter and L.J.Th.Van der Kamp, eds, John Wiley & Sons, NY, 1976, pp. 135–157.
L. Linn and J.A. Slindle, The determination of the significance of change between pre- and posttesting periods, Review of Educational Research 47 (1977), 121–150.
F.M. Lord, The measurement of growth, Educational and PsychologicalMeasurement16 (1956), 421–437.
S. Maxwell, H.D. Delaney and J. Manheimer, ANOVA of residuals andANCOVA: Correcting an illusion by using model comparisons and graphs, Journal of Educational Statistics 95 (1985), 136–147.
G.J. Mellenbergh, A note on simple gain score precision, AppliedPsychologicalMeasurement 23 (1999), 87–89.
J.E. Overall and J. A. Woodward, Unreliability of difference scores: A paradox for measurement of change, PsychologicalBulletin82 (1975), 85–86.
D. Rogosa, D. Brandt and M. Zimowski, A growth curve approach to the measurement of change, Psychological Bulletin 92 (1982), 726–748.
I. Rop, The application of a linear logistic model describing the effects of preschool education on cognitive growth, in: Some D.M. Dimitrov and P.D. Rumrill, Jr. / Pretest-posttest designs and measurement of change 165 mathematical models for social psychology, W.H. KempfandB.H. Repp, eds, Huber, Bern, 1976.
D. S¨orbom, A statistical model for the measurement of change in true scores, in: Advances in Psychological and Educational Measurement, D.N.M. DeGruijter and L.J.Th. Van der Kamp, eds, John Wiley & Sons, NY, 1976, pp. 1159–1170.
58 | Page