Design rubric to evaluate pedagogical practices (in preess)

Design and Validity of a Rubric to Evaluate Pedagogical Practices with a Socioformative Approach Sergio Tobón*, Julio H. Pimienta-Prieto**, Luis Gibran Juárez-Hernández** and José Silvano Hernández-Mosqueda** **CIFE University Center, Cuernavaca, Morelos, Mexico Ekap University, Florida, USA E-mail: stobon@cife.edu.mx ** CIFE University Center, Cuernavaca, Morelos, Mexico (www.cife.edu.mx)

In press: Tobón, S., Pimienta-Prieto, J. H., Juárez-Hernández, L. G., & Hernández-Mosqueda, J. S. (2018). Design and Validity of a Rubric to Evaluate Pedagogical Practices with a Socioformative Approach. Information, in press.

Abstract The purpose of this study was to design and validate a rubric to evaluate the pedagogical practices of elementary school teachers in Mexico. The instrument was reviewed by three experts; then, the content validity study was carried out by 21 judges through the application of Aiken’s V, resulting in values higher than 0.80. once the content validity was demonstrated, the rubric was applied to 510 teachers working in the State of San Luis de Potosí, Mexico, during the first semester of 2018. The factor analysis showed that the rubric evaluates a single factor, as suggested at the theoretical level. Finally, the reliability, measured with Cronbach’s Alpha coefficient was 0.90. It is concluded that the rubric designed to evaluate the pedagogical practices of elementary education teachers in Mexico has adequate levels of validity and reliability.

Key Words: Pedagogical practices, Socioformative rubrics, Socioformation

1. Introduction Society has changed into an information society and the challenge is to reach a knowledge society, which involves transforming communities and organizations to work collaboratively in the improvement of living conditions by managing and co-creating knowledge in various sources, based on information and communication technologies. To achieve this change, it is proposed that in Latin America, socioformation, which is an approach with the aim of forming citizens, teams and communities by solving context problems and contributing to improve living conditions, based on an ethical life project, entrepreneurship, collaborative work, the cocreation of knowledge, metacognition and complex thinking [1].

In socioformation, one looks to transform traditional and improper pedagogical into suitable pedagogical practices that help human and social development, promote inclusion [2] and creativity [3]. Pedagogical practices are defined as actions that are executed with the students

to achieve the expected learning through various teaching strategies and learning resources [4].

They have an impact on student learning [5] and they need to be evaluated systematically.

Various instruments for evaluating pedagogical practices have been proposed [6, 7, 8], but they have the following voids: 1) they do not consider the challenges of a knowledge society nor socioformation; 2) they are generally based on checklists or scales that do not allow assessing the performance level of the practices [9]; 3) many of the instruments have been developed in contexts different from that of Latin America; and 4) they address educational activities, but not transformative pedagogical practices.

Therefore, the goals of the present study are: 1) to design a rubric to evaluate the transformative pedagogical practices of teachers with a socioformative approach through the support of experts; 2) to determine the content validity of the designed rubric; 3) to determine the reliability of the rubric through its use in a group of teachers; and 4) to establish the validity of the instrument based on factor analysis.

2. Materials and Methods The study of validity and reliability of the instrument was carried out by means of the following phases:

2.1 Design and review by experts A rubric was developed to evaluate the pedagogical practices with a socioformative approach with the support of three experts in the area. The experts were researchers in mediation and learning processes with at least ten years of experience in the area and at least five articles published in scientific journals on the subject. The rubric was composed of ten indicators, which are described in Table 1. Each indicator is composed of five descriptors that follow the socioformative taxonomy [1]: Pre-formal Level (very low), Receptive Level (low), Resolute Level (medium), Autonomous Level (medium high) and Strategic Level (very high). For the present study, a series of sociodemographic questions were added, such as age and gender.

Table 1. Practices evaluated by the Rubric on Pedagogical Practices Practices Practice 1. Motivation and achievement of expected learning in students Practice 2. Learning concepts through organizers and cases Practice 3. Problem solving

Practice 4. Formation of universal values and an ethical life project Practice 5. Assertive communication

Description To what degree do you motivate students and focus on achieving their expected learning?

To what extent do you approach the learning of concepts through motivating strategies?

To what extent do you address the resolution of context problems, that is, the resolution of real needs, such as family, community, science, health, etc.?

To what extent does it address the formation of values in the classroom and stimulate personal growth?

To what degree do you approach assertive communication in the classroom?

Practice 6. Collaborative work Practice 7. Development of creativity Practice 8. Application of mainstreaming Practice 9. Resource management Practice 10. Formative and metacognitive evaluation

To what degree do you teach students collaborative work in class?

To what degree do you teach students to have creativity and innovation in what they do?

To what degree do you teach your students to solve problems with mainstreaming, that is, articulating knowledge from various disciplines?

To what degree do you teach students how to manage resources from different sources to solve problems?

To what degree do you use evaluation for continuous improvement in your students?

2.2 Application of the test to a pilot group The test was applied to a pilot group composed of 25 teachers. This allowed to improve the clarity of the instructions and questions, and to identify the suitability and clarity of the questions (Table 2).

Table 2. Demographic data of the pilot group Gender (%)

Women: 88% Men: 12%

Age (median + standard deviation)

30.44 (+ 5.76)

Years of experience as a teacher (median + standard deviation)

7.2 (+ 5.97)

2.3 Content validity study The instrument was improved from the pilot group and then subjected to a content evaluation by 21 judges (Table 3 describes their qualifications). They assessed the suitability and the wording in four degrees: 1-4, with 3 being the acceptable level and 4 the level of excellence or higher. The degree of satisfaction with the instrument was also assessed on a scale of 0 to 5, with 5 being the highest level of satisfaction. From this, the content validity was evaluated by means of Aiken’s V method [10]. The criterion for accepting the items as valid was that they presented a V higher than 0.80.

Table 3. Data about the qualifications of the expert judges N

21 judges

Gender (%)

Women: 71% Men: 29%

Age (mean + standard deviation)

35.66 (+ 7.45)

Last degree of study (%) Areas of professional experience Years of experience as a teacher (mean + standard deviation) Number of hours of didactic training and evaluation in the last two years (mean + standard deviation)

Master: 90% PhD: 5% Postdoctoral PhD: 5% Teaching: 100% 10.47 (+ 7.36) 274.7 (+ 598.3)

Average number of articles published in the field (mean + standard deviation)

1.42 (+ 1.56)

Average of books published in the field (mean + standard deviation)

0.42 (+ 0.50)

Average of book chapters (mean + standard deviation)

1.14 (+ 2.41)

Average number of lectures (mean + standard deviation) Percentage of experts with experience in reviewing, designing and/or validating a specific research instrument

0.80 (+ 0.74) 100%

2.4 Evaluation of suitability and satisfaction Based on the judgment of the experts, some improvements were made on the indicators and these were applied to a sample of 510 elementary school teachers, who had just entered the professional teaching service, and who were available and accepted to participate in the study (Table 5). The application of the rubric was conducted online. They were asked the same questions that the experts were asked, about the suitability, wording and satisfaction with the instrument.

Table 4. Sociodemographic data of the target population n

510

Gender

Women: 80.40% Men: 19.60%

Age (mean + standard deviation)

29.73 (+ 7.42) Single: 55.5 %

Marital status

Married: 41.37% Divorced: 2.35%

Residence area Economic conditions (grade: 0-5, where 0 = Very low conditions, and 5 = excellent economic conditions) (mean + standard deviation) Years of experience as a university teacher (mean + standard deviation)

Widow/widower: 0.78% Rural: 13.72% Semi-urban: 11.38% Urban: 74.9% 2.09 (+ 0.59) 0.35 (+ 2.79)

2.5 Factor analysis With the data from the same sample of 510 teachers, the construct validity analysis was performed. It was carried out through an exploratory factor analysis (AFE) (Table 4). To verify the adequacy of the data to the AFE, the correlation matrix was inspected, and the KMO (Kaiser Meyer Olkin) index and the Bartlett sphericity test were used. If the data were suitable for analysis by the EFA, the main axis factorization extraction method was chosen and the number of components to be retained was based on the Gutman-Kaiser rule [11]. The main axis factorization extraction method was chosen because it is a robust method to face possible violations of the assumption of normality and allows moderate correlations [12]. If the factorial loads did not present a single factor load, a rotation with the Varimax algorithm was made.

2.6 Reliability study Finally, the internal consistency of the instrument was determined by Cronbach's Alpha coefficient, after assessing compliance with the requirements [13].

3. Results 3.1 Application to a group of experts and pilot study

The rubric was submitted to a group of three experts who made a series of suggestions to improve the wording of the descriptors. The pilot group of 25 basic education favorably assessed the suitability and structure of the questions, and the average was 3.0 or higher (acceptable level). Regarding satisfaction with the instrument, the degree was medium high, as it was 4.0 out of 5.0 (Table 5).

Table 5. Assessment of the instrument by the pilot group n

25

Items Suitability of the questions to evaluate essential pedagogical practices (degrees: 1-4).

Wording of the questions and of the answer levels for each question (degrees: 14) Satisfaction with the pedagogical practices questionnaire (degrees: 0-5)

Mean (+ standard deviation) 3.0 (+ 0.76) 3.48 (+ 0.50) 4.0 (+ 0.91)

3.2 Content validity Table 6 describes the values of Aiken’s V obtained from the application to the group of 21 judges. It can be seen that all values were greater than 0.8, which shows that it has content validity.

Table 6. Results of content validity n

21

Items Suitability of the questions to evaluate essential pedagogical practices.

Wording of the questions and of the answer levels for each question Satisfaction with the pedagogical practices questionnaire

Mean 3.5714 3.5714 4.2380

Aiken’s V 0.85714 0.85714 0.80951

3.3 Suitability and quality Table 7 describes the descriptive data of the instrument, obtained from the application to a sample of 510 teachers. Table 7 shows the assessment of suitability, the level of wording of the rubric and the degree of satisfaction with the instrument by the teachers evaluated. It can be seen that the instrument has good values for suitability, wording of the indicators and descriptors and degree of satisfaction by the teachers themselves, as they are higher than 3.0, which is the Acceptable level.

Table 7 Assessment of the instrument by the target population n Items Suitability of the questions to evaluate essential pedagogical practices.

Wording of the questions and of the answer levels for each question Satisfaction with the pedagogical practices questionnaire

510 teachers Mean (+ standard deviation) 3.18 (+ 0.76) 3.32 (+ 0.61) 3.48 (+ 0.90)

3.4 Construct validity and reliability The correlation matrix shows statistically significant correlations between all the items evaluated (Table 8). The Bartlett test (Xi: 1.676, p <0.001) and the KMO Index (0.914), indicated that the data can be analyzed by the AFE.

Table 8. Correlation matrix Practice 1

Practice Practice Practice Practice Practice Practice Practice Practice Practice Practice 1 2 3 4 5 6 7 8 9 10 1.000

Practice 2

.496*

1.000

Practice 3

.333*

.397*

1.000

Practice 4

.375*

.356*

.393*

1.000

Practice 5

.445*

.391*

.374*

.486*

1.000

Practice 6

.391*

.369*

.401*

.410*

.450*

1.000

Practice 7

.351*

.422*

.328*

.344*

.385*

.452*

1.000

Practice 8

.371*

.410*

.285*

.331*

.351*

.394*

.506*

1.000

Practice 9

.356*

.428*

.356*

.390*

.377*

.367*

.482*

.531*

1.000

Practice 10

.467*

.421*

.372*

.399*

.419*

.402*

.320*

.379*

.441*

1.000

Exactly, the communalities presented adequate values, indicating their representation in the factorial model (Table 9). In this respect, the analysis produced a one-factor solution, which explained more than 45% of the variance. This factor presented an initial eigenvalue of 4,585, and included the ten proposed items (with loads greater than 0.50) (Table 10), coinciding with the theoretical proposal of the construct addressed.

Table 9. Communalities Practices Practice 1

Initial

Taking .382

.400

Practice 2

.389

.425

Practice 3

.295

.318

Practice 4

.348

.373

Practice 5

.388

.421

Practice 6

.372

.410

Practice 7

.397

.402

Practice 8

.400

.395

Practice 9

.420

.435

Practice 10

.373

.407

Table 10. Factorial matrix Factors: 1 Practice 1

.633

Practice 2

.652

Practice 3

.564

Practice 4

.611

Practice 5

.649

Practice 6

.640

Practice 7

.634

Practice 8

.629

Practice 9

.660

Practice 10

.638

Finally, the instrument presented a high reliability of 0.90, measured by Cronbach's Alpha.

4. Discussion The study that was carried out presents evidence that the Rubric of Pedagogical Practices with a socioformative approach (SOCME-10) has adequate levels of content validity, based on the application of Aikenâ&#x20AC;&#x2122;s V, inasmuch as the values were higher than 0.8 for the three variables evaluated by the group of experts in the field: suitability, wording of the descriptors and degree of satisfaction. The judges were teachers with experience in didactics and learning assessment processes, with training in the socioformative approach and experience in the design of evaluation instruments. Regarding the construct validity, only one factor was found. This is in line with what is expected at the theoretical level, as the pedagogical practices emphasize one dimension, such as the work of the students with problem solving and contribution to society.

It was also found that a reliability of 0.90 is considered a high value [13]. Finally, it was found that the rubric was evaluated by the same users with appropriate levels of suitability, wording and satisfaction, given that the average of the three aspects evaluated was higher than 3.0, which was within the Acceptable level. The latter is important in the construction of instruments [14] because evaluation tools are often designed with high levels of validity and reliability, but these are not always suitable for those who will utilize them. It is concluded, then, that the presented instrument is suitable and has adequate levels of validity and reliability.

In the literature, several instruments to evaluate pedagogical practices in teachers are reported [6, 14]. However, these are not focused on practices that can impact and transform education, but rather on the activities that every teacher, in general, must implement. The rubric validated in the present study has the main novelty of focusing on those actions that are essential to form citizens of a knowledge society, such as problem solving, collaboration, metacognitive evaluation, values and an ethical life project, mainstreaming and creativity, etc. Furthermore, this new rubric presents several innovative aspects, such as: 1) the application of socioformative taxonomy in the development of descriptors, which provide teachers with a more precise and detailed tool that makes it possible for them to evaluate themselves step by step with regard to their pedagogical practices and, in this way, improve, which is essential in this field [15]; 2) each of the ten pedagogical practices evaluated in this rubric assesses five levels of performance, and they range from a traditional, content-centered practice to a more innovative instructional practice, according to the challenges of today and those of the near future; and 3) the promotion of the formation of people for a sustainable social development, a key goal in communities all over the world, towards which education should be oriented, and which is hardly ever evaluated in current instruments of pedagogical practice.

The present study is exploratory and must be confirmed by other studies that apply the same

instrument in similar populations. Moreover, it is necessary to determine other aspects of the validity of the instrument, such as consistency over time, and the concurrent and divergent validity of the components with other instruments or items that have an already proven validity with regard to pedagogical practices. It is also important to determine the validity and reliability in teachers that work in higher levels of education, since the present study was carried out in basic education.

References [1] Tobón, S., Conceptual analysis of the socioformation according to the knowledge society.

Knowledge Society and Quality of Life (KSQL), 1 (2017), 9-35. https://goo.gl/aJeSvw [2] Castro-Rubilar, F., Castañeda-Díaz, M. T., Ossa-Cornejo, C., Blanco-Hadi, E., & CastilloValenzuela, N., Validación de la escala de autoadscripción inclusiva en docentes secundarios de Chile.

Psicología Educativa, 23 (2017), 105-113. https://doi.org/10.1016/j.pse.2017.05.003 http://www.sciencedirect.com/science/article/pii/S1135755X17300209 [3] Karwowski, M., & Lebuda, I., Creative self-concept: A surface characteristic of creative personality. In G. Feist, R. Reiter-Palmon, & J. C. Kaufman (Eds.), Cambridge handbook of creativity and personality research. New York: Cambridge University Press, 2016.

[4] Álvarez-Álvarez, C., Teoría frente a práctica educativa: algunos problemas y propuestas de solución. Perfiles Educativos, 37 (2015), 172-190. https://doi.org/10.1016/j.pe.2015.11.014. http://www.sciencedirect.com/science/article/pii/S0185269815000264 [5] Chrispeels, J. H., Andrews, C. A., & González, M., System Supports for Teacher Learning and School Improvement. In T. Townsend (Ed.), International Handbook of School Effectiveness and Improvement (Vol. 2, pp. 787-806). Netherlands, Springer, 2007.

[6] Pianta, R., Hamre, B., & Mintz, S., Classroom Assessment Scoring System. Upper Elementary Manual. Teachstone, Charlottesville, Virginia, 2012.

[7] Seidman, E., Raza, M., Kim, S., & McCoy, J. M., Teacher instructional practices and processes system (V.5) e TIPPS: Manual and scoring system. New York University, 2013.

[8] UNESCO, Overview Measuring Early Learning Quality and Outcomes. Paris, UNESCO, 2017. http://unesdoc.unesco.org/images/0024/002480/248053e.pdf [9] USAID, Data collection monitoring report: Standards-based classroom observation protocol for Egypt (SCOPE) in GILO and TILO schools. Egypt, USAID, 2010. Retrieved from the GEM2 website http://www.gem2.org/node/190.

[10] Penfield, R.D., & Giacobbi, Jr, P.R., Applying a score confidence interval to Aiken item content-relevance index. Measurement in Physical Education and Exercise Science, 8 (2004), 213-225.

[11] Gorsuch, R. L., Factor analysis. Hillsdale, NJ, Lawrence Erlbaum, 1983.

[12] De Winter, J. C., & Dodou, D., Factor recovery by principal axis factoring and maximum likelihood factor analysis as a function of factor pattern and sample size. Journal of Applied Statistics, 39 (2012), 695-710.

[13] Cronbach, L.J., Coefficient alpha and the internal structure of tests. Psychometrika, 16 (1951), 297- 334.

[14] Seidman, E., Kim, S., Raza, M., Ishihara, M., & Halpin, P. F., Assessment of pedagogical practices and processes in low and middle income countries: Findings from secondary school classrooms in Uganda. Teaching and Teacher Education, 71 (2018), 283-296. https://doi.org/10.1016/j.tate.2017.12.017. http://www.sciencedirect.com/science/article/pii/S0742051X17303402 [15] Seidman, E., An emerging action science of social settings. American Journal of Community Psychology, 50 (2012), 1e16. https://doi.org/10.1007/s10464-011- 9469-3, 0:1e16.

*Corresponding author: Sergio Tobรณn, Ph.D. CIFE University Center, Cuernavaca, Morelos, Mexico. www.cife.edu.mx Ekap University, United States. www.ekap.us E-mail: stobon@cife.edu.mx