The effectiveness of training teachers in problem-based learning implementation on students’ outcomes: a mixed-method study

Study design

A quasi-experimental design was adopted in this study as the main quantitative approach to minimise bias in estimating the difference between traditional instruction and PBL classes. In addition, a qualitative approach was used during the intervention using field observation notes and after the intervention using interviews, as a secondary approach (see Fig. 1).

The figure illustrates the study design; mathematical test and attitudes towards mathematics were applied before and after the intervention, while during the quasi-experimental implementation, field observation notes were taken, and at the end of the intervention semi-structured interviews were conducted with the teachers.

Figure 1 illustrates the study design; during the quasi-experimental implementation, field observation notes documenting the authors’ observations were taken with the aim of observing how teachers implemented PBL, while semi-structured interviews were conducted with both types of the teachers who only implemented PBL (trained and untrained teachers) after the implementation of PBL as a supplement, with the aim of being used as part of the triangulation method for the author’s observations in how teachers implemented PBL.

School and participating students

The school was located in an urban district in a major city, Hail, which is situated in the north of Saudi Arabia. The school was randomly selected from ten private schools. Then, seven of the third-grade classes out of nine in the selected school were randomly chosen. The third grade is an important level, as it is the final grade of lower primary school. The classes were instructed by three teachers; one taught three classes, and the others taught two classes each. These classes comprised the three following groups: group A (three classes taught by a trained teacher using a PBL teaching strategy), group B (two classes taught by an untrained teacher using TTM) and group C (two classes taught by an untrained teacher using a PBL teaching strategy; see the study design in Table 1).

Ethical approval was obtained, and all participants signed consent forms to participate. They were informed that they could withdraw any time with no need to justify their decision, nor would there be any consequences of withdrawal.

In total, 127 pupils participated in the study, and their ages ranged from eight to nine years old. They were in the last semester of the third grade. Most of the students at the school were Saudis; in each group, two to four students had Arab backgrounds, such as from Syria, Egypt and Sudan. All students had a middle-class socioeconomic status. Academic school records and pre-test’ scores were used to ensure that the groups were similar in terms of mathematical achievement. Within each group, students showed a wide range of academic achievements; the students spanned from very low to very high achievers. There were no special education pupils within the groups.

Teachers

Three teachers were randomly selected from one large primary school to take part in this study. The first teacher was randomly selected to receive training courses in using the PBL teaching strategy. The second teacher did not receive any training, but he was provided with PBL materials—specifically, design problems and guidelines for implementing PBL; he was asked to conduct self-directed learning (SDL) to implement PBL in his classrooms. The aim of including a trained and an untrained teacher using PBL was to measure the effects of training teachers on students’ outcomes. The third teacher was not trained in PBL and was asked to teach students using TTM.

The teachers had similar characteristics in terms of qualifications, experience and expertise, as well as in their beliefs and perspectives on PBL and TTM. They are all male and they believed that the aim of teaching mathematics is to conduct real-life problem solving, and they considered active learning to be important for students. They had been teaching mathematics to third-grade school students for 10 years. They all had a first degree in mathematics. They were all Egyptians and aged in their late thirties. According to the teachers and the administration of the school, the teachers had all attended the same training courses in different aspects of education, such as active learning. However, none of them had ever been trained in using PBL teaching strategies.

Topics

The topic covered in the classes was ‘data display’. It covered representation through codes, interpretation of representation through codes, representation in columns and interpretation of representation in columns. The content was new to the students. The instruction took place during ten class sessions (45 min each) comprising four sessions per week over for two and a half weeks, with a total of 7.5 h for each group. To control for the time factor, all groups, whether PBL or traditional, were given the same amount of time.

Instruments

Six multiple-choice questions, short answer questions, fill-in table questions and drawing tests were applied at the beginning of the study (pre-test) and in the final experiment (post-test). Mathematics items were selected from Trends in International Mathematics and Science Study (TIMSS) 2003, 2007 and 2011 (see Mullis et al., 2012). The TIMSS items that were selected matched the objectives of lessons for knowledge application exactly; they had already been examined for the purpose of the test. We chose TIMSS mathematics items because they were verified as appropriate for the students’ ages. The students had nearly finished the third grade, and the curriculum for that grade contained many TIMSS topics (see TIMSS, n.d.). Each item on the test received a score of either one or zero. An example of the items is given in Appendix A. The measure ‘attitudes towards mathematics’ of TIMSS 2007 (Mullis et al., 2008) contains four items, as follows:

1. 1.

   I would like to take more mathematics in school

2. 2.

   I enjoy learning mathematics.

3. 3.

   Mathematics is boring (reverse-coded).

4. 4.

   I like mathematics.

This measure was adopted and assumed to meet the standard of a valid and reliable test (see, Mullis et al., 2008). Attitudes were assessed using four items applied twice as pre- and post-measures; four items with 4-point Likert scales (disagree a lot, disagree a little, agree a little, and agree a lot) were presented. Each item score ranged from 1 to 4. The total marks ranged from the number of items of the measure to multiply them by 4; the measure consisted of four items, so the total scores ranged from 4 to 16. Some items were reverse-coded; for example, for ‘mathematics is boring’, ‘disagree a lot’ would receive a score of 4, whereas ‘agree a lot’ would receive a score of 1.

The face validity method was used to assess the validity of the tests and attitude measures. Eight arbitrators checked and gave their opinions on the adequacy, clarity, and relevance of the items’ content. The opinions of the arbitrators were considered and included in the preparation of the final image of the tests and attitudes. However, no changes were reported, and face validity confirmed the tests’ validity. In addition, test-retest reliability was used to assess the reliability of the tests and attitude measures. The levels of reliability were acceptable, with a score of 0.86 for the mathematics test and 0.88 for the attitude measure. For further reliability, Cronbach’s alpha was used for each scale of the test and attitudes and for the whole test and attitudes. The results show that all items correlated with a good degree of total scales (no items scored less than 0.3), and the reliability for the test was 0.747, whereas that for attitude was 0.808. Therefore, the measures became high valid for the purposes of this study.

In qualitative methods, filed observation and semi-structured interview were used to assess teachers’ performance in PBL implementation. After filed observations completed, post- semi-structured interviews were conducted for the teachers to confirm the results of author observations of how teachers implemented PBL as a supplement for the methodological triangulation of the filed observations. Methodological triangulation involves a researcher using more than one method, such as interviews and observations, for collecting data to understand a phenomenon deeply (Flick et al., 2004; Neuman, 2000). The teachers’ responses to the questions in the semi-structured interviews were analysed and compared with the analysed observation data to enhance the validity of the study and to gain a deeper understanding of social events. As Neuman (2000) commented, “Looking at something from several different points gives a more accurate view of it” (p. 521).

The data obtained from qualitative methods were deductively analysed. Prior to conducting data collection from filed work. A structured categorisation matrix was developed by the authors based on a literature review (see Barrows, 1998; English and Kitsantas, 2013; Hmelo-Silver and Barrows, 2006, 2008). It aimed to assess PBL implementation conducted by teachers and consisted of two main categories: understanding the problem and using metacognitive strategies (see Appendix B). Field observation notes were used to describe how the teachers implemented PBL. In this study, field observation notes consisted of two parts: descriptive and reflective information (Patton, 1990). The descriptive part involved documenting the factual data obtained from inside the classroom. The main author moved between groups to make sure everything was proceeding well; the intention was to monitor the implementation of the study, and the authors had a diary that was used to document any observations, particularly the observations that took place during lessons and were made inside mathematics classrooms. The main focus was on teachers’ performance, particularly with respect to teacher intervention, individual and collective student practices, student responses, group interaction and PBL processes. In the reflective section, the authors reflected on the meaning of the observations outside of the classroom (see Appendix C). At the end of the experiment, ten lessons by each teacher were observed.

Semi-structured interview questions were developed according to analysed data of class observations which includes: The three main questions:

1. 1.

   How was PBL implemented in your teaching strategies?

2. 2.

   How did you assess your students in relation to understanding the problem?

3. 3.

   How did you support your students to solve the problem?

In semi-structured interview, tape recordings were used for the interviews with each teacher, which ranged from 13 to 23 min in length. The interviews were conducted in Arabic, transcribed and subsequently translated into English by the authors.

The data were deductively coded (i.e., both the interview and observation) by the main author, and according to the identified categories mentioned above. When a deductive content analysis is used, a categorisation matrix is developed; following this, the data are coded according to the categories (Polit and Beck, 2004). In addition, if a structured matrix is chosen, only aspects that fit the matrix are selected from the data (Patton, 1990).

Professional development

The PBL programme used in this study aimed to train teachers by focusing on how to implement PBL in mathematics classrooms. The programme continued to provide feedback during the implementation after each session, taking advantage of the literature recommendations. Therefore, the trained teacher learned how to facilitate groups’ learning processes and guide students’ learning by adopting strategies such as posing meta-cognitive questions and focusing on the process of learning to model students’ learning strategies. The teacher was trained in intervention strategies, such as making decisions based on what, when and how intervention should occur to enhance cooperation. The programme included examples of PBL implementations. Teacher training lasted for one week (8–10 h), and daily meetings took place during the course of the training to provide an opportunity to present feedback and resolve unexpected problems. The programme for training the teacher to implement PBL in his class was developed by the author. It was expected that, following the teacher’s completion of the programme, the teacher would be able to do the following:

• provide scaffolding and feedback as needed

• prompt independent thinking

• facilitate collaborative knowledge construction for students

• monitor learning processes

• model desired behaviours

• concentrate students’ efforts on critical thinking.

• use intervention strategies, such as making decisions on what, when and how to intervene

The programme included three real-life sessions, each lasting 45 min. The teacher was asked to implement the PBL strategy using an ill-structured problem, which was taken from a mathematics textbook and related to the topics that the students had been studying. A group of students from outside the study sample was selected to assess the teacher’s performance and establish whether he was able to implement PBL effectively. This was followed by providing the teacher with extensive feedback, which lasted more than an hour for each session.

The students were trained in two sessions in how to deal with the PBL teaching strategy.

Problem-based learning implementation

Problems were presented to the students. Students worked in small groups of four to six members. They discussed their understanding of the problems, and then the teacher discussed the understanding of the problem with the whole class. This was followed by students solving the problems. Finally, the teacher discussed the solution with all the students.

In this study, the six core characteristics of PBL mentioned by Barrows (1996) were adopted. These are as follows:

1. 1.

   The student is the centre of the learning.

2. 2.

   Learning occurs in small groups of students.

3. 3.

   At the beginning of the learning, the students are presented with authentic problems.

4. 4.

   The problems are used as a means of developing problem-solving skills.

5. 5.

   New knowledge is gained through SDL. (Barrows, 1996)

6. 6.

   The student is the centre of the learning.

From the literature review (see Barrows, 1986; Gallagher and Stepien, 1996; Hung et al., 2008), six characteristics were adopted in the problems after reviewing the literature related to the problem of PBL. These were as follows:

Statistical analysis (quantitative analysis)

The study used mixed-factor analysis of variance (ANOVA) models (Field, 2013; Howell, 2012) within one factor (time: pre- and post-tests and between). Tukey’s post hoc test (Field, 2013; Howell, 2012) was applied when appropriate and where significant results were observed—that is, an effect size (partial eta squared [η_p²]). The effect size, classified as Cohen suggested, could be small 0.01; medium, 0.06; or large, 0.14. All analyses were performed on IBM SPSS v22 and at a 5% (0.05) level of significance.

Procedure

A quasi-experimental design was adopted in this study as the main quantitative approach, while a qualitative approach was used during the intervention using class observation notes and interviews, as a secondary approach. In total, 127 pupils participated in the study. They were in the last semester of the third grade. Ethical approval was obtained, and all participants signed consent forms to participate. Three teachers were randomly selected from one large primary school to take part in this study. The first teacher was randomly selected to receive training courses in using the PBL teaching strategy. The second teacher was not trained and asked to conduct SDL to implement PBL in his classrooms. The third teacher was not trained in PBL and was asked to teach students using TTM. The topic covered in the classes was ‘data display’. The content was new to the students. The instruction took place during 10 class sessions. Instruments of the study include mathematics test and attitudes towards mathematics were prepared and verified. Applying a pre-test (a measure of attitudes towards mathematics and an exam to measure mathematics application). Conducting the study took about 2 and a half weeks. Applying for a post-test (a measure of attitudes towards mathematics and an exam to measure mathematics application). During the intervention, class observations were carried out for each lesson.