The Effects of Online Problem-Based Learning Methods on Students’ Performance

The COVID-19 pandemic has a significant impact on the education process in higher education institutions. The medical faculty is forced to adapt by implementing distance learning. The entire face-to-face learning is replaced with virtual meetings to prevent transmission of the COVID-19 virus. Problem-based learning (PBL) is carried out online. This study aimed to analyse students' performance during an online PBL in terms of their content knowledge about the modules. The study was conducted in the 2019/2020 academic year with 92 respondents enrolled in the Neuropsychiatry course. The students’ performance was measured by direct observation using a rubric filled out by the tutor during the session. Content knowledge about the module was measured by test, before and after the PBL. A cross-sectional study design was used in this study. The results showed a statistically significant increased score between pre and post-test (with p-value < 0.000, Wilcoxon signed-rank test) with an overall better performance of the students during the online PBL. Students showed good participation and communication skills, even though the discussions Ulum Islamiyyah Journal | Special Issue: Healthcare in Pandemic Era: “The New Norm” Vol.3 74 were done synchronously via teleconferencing. They had the ability to apply and link concepts to problems and draw valid conclusions. As a conclusion, there was a weak association between students’ performance and their knowledge of the module contents shown by the increasing score in the test. The limitations or obstacles during online PBL can be overcome with better preparation from the students with sufficient digital literacy, self-regulation, and adequate internet network support.


Introduction
The student-centred, problem-based, integrated, community-based, elective, systematic (SPICES) approach has been applied in almost all medical higher education institutions with various learning models. The implementation of SPICES is done conventionally, online or using a combination of both, known as blended learning. There is a recent trend towards small-group work in undergraduate medical courses. The current understanding of educational strategies supports the use of small groups as an effective method of learning. Small-group work recognizes a movement towards learner-centred, problem-based, and self-directed learning. A variety of methods can be used with small groups; one of them is tutorial.
The increasing number of COVID-19 cases in Indonesia has forced educational institutions to organize learning processes online. This is done to reduce the spread of the virus. During the pandemic, in medical education, adaptation in learning was carried out in various ways, including problem-based learning (PBL). Online PBL was implemented through asynchronous and synchronous teleconferencing with tutors. In facing the challenges of online learning, our medical faculty members made adaptations of learning methods. Tutorial discussions are still conducted with tutor guidance via the online system using the zoom meeting application. Forms are provided via Google Form which contain a rubric for evaluating students' performance in the tutorial. For supporting online learning activities, good internet network is needed along with a discussion platform that can be implemented synchronously and asynchronously, and the ability of lecturers and students to navigate the platform. Previous research shows that online PBL had a significant effect on increasing critical thinking skills. An important relationship between discussion and problem solving was found during online PBL sessions (Şendağ & Ferhan Odabaşı, 2009). In another study with the integration of online and in-person design of PBL, it was reported that students transferred what they had learned from online discussions to their actual discussions during problem-solving (Cho & Jonassen, 2002).
It is also crucial to understand the level of student engagement during the online learning process. The retention rate of online learning engagement is reported to be lower than that of direct face-toface learning (Dietz-Uhler et al., 2007). Based on this, the objective of this study was to analyse students' performance during an online PBL session in terms of their content knowledge about the modules. The performance was measured by direct observation with rubric filled out by the tutor and their level of understanding about the module contents, cognitive abilities, pre, and post-test, before and after the PBL session.

Study Design
The research method used was analytic cross-sectional. It was conducted on the 4 th semester, involving undergraduate medical students enrolled in the Neuropsychiatry course.

Respondents
The participants were students enrolled in the Neuropsychiatry course provided by the Faculty of Medicine, University of Muhammadiyah Makassar in the 2019/2020 academic year. This study used nonprobability sampling, that is total sampling. Total sampling is a sampling technique when all members of the population are used as a sample. Based on the sampling technique, all students enrolled in the Neuropsychiatry course became the samples, totalling 92 respondents. The inclusion criterion was a minimum attendance of 80% in the course, and the exclusion criterion was students not fully present in the tutorial process.

Module
Modules used in the Neuropsychiatry course consist of 2 modules; headache and hemiparesis. The topic of the module is based on a list of competencies for general practitioners listed in the Indonesian Doctors Competency Standards (SKDI) in 2012 (Indonesia, 2012). These modules are equipped with scenarios, learning strategies, student assignments, guidance for tutors, several alternative questions and answers, and several references. Each module consists of four different scenarios. These modules were given to the students via Learning Management System (LMS) Moodle, known as SPADA (Sistem Pembelajaran Daring) at site www.spada.unismuh.ac.id. (

The implementation of online tutorial group discussion
The neuropsychiatry course ran for six weeks. The implementation of the tutorial group discussion in the second week preceded expert lectures. Students were then divided into small groups with facilitated tutors. Students worked in small groups for 2x50 minutes, and they had some discussions about the module. The tutorial group discussion consisted of four meetings. In the first meeting, students discussed the scenario to determine various possible diagnoses sorted by their level of possibility. Students conducted independent learning in the second meeting to look for additional information needed to establish a temporary diagnosis. In the third meeting, students returned to the discussion by presenting the findings and establishing a brief diagnosis. Each group had different diagnostic conclusions. The fourth meeting was a plenary session, where every group showed their results. In this plenary session, students could get feedback and additional information from the experts. All the meetings in the tutorial session were delivered synchronously by teleconferencing using Zoom (Figure 3). In PBL, students were assessed based on actual performance in their learning setting. Students were evaluated in terms of their ability to make relevant hypotheses, identify their learning needs, use appropriate learning resources, provide evidence of learning, and act as a responsible team member in the tutorial group (Valle et al., 1999). Based on that, we developed instruments that can evaluate those abilities, including (1) ability to respect others during the discussion, (2) contribution in discussion, (3) quality of information, (4) ability to define and formulate problems, (5) ability to make a relationship from various data/facts and (6) ability to analyze and synthesize data/facts. A 100-point grading system was used, where the total score represents "unsatisfactory" (< 60), "good" (60 -80), and "outstanding" (> 80)" (Figure 4).

Pre and Post Test
To evaluate the level of understanding about the modules contents, a pre-post test was conducted, with the same questions. The pre-test was conducted before the first meeting, and the post-test was conducted after the plenary session. This test was conducted to assess students' cognitive abilities with multiple choice questions (MCQ). Each test consisted of 10 questions, comprising all materials related to the module which have been taught during the course. The tests were closed-book tests, and done by computer-based tests (CBT) with learning management system SPADA (Sistem Pembelajaran Daring) in an synchronous session with proctoring using zoom.

Statistical Analysis
Kolmogorov-Smirnov normality test was carried out, and the data is not normally distributed. To determine the median difference between pre and post test scores, a Wilcoxon signed-rank test was used. Spearman's correlation was employed for determining the relationship between students' performance and the average total test scores.

Results
Data on the students' performance during an online tutorial session can be seen in Table 1. These results revealed high overall performances of the students during the online tutorial (Table 1).  For this numerical data, the Kolmogorov-Smirnov normality test was carried out, and the data were not normally distributed. A Wilcoxon signed-rank test was conducted to determine whether there was a statistically significant median difference between the pre-test compared to post-test score. Data were shown as mean ± standard deviation and as median (interquartile range). In the test for module I (headache), from the 92 students, 57 students had improvement, whereas 19 students had no improvement, and 16 students showed decreasing results for the test. Meanwhile, in the test for module II (hemiparesis), 78 students showed an improvement, only three students had no improvement, and 11 students showed a decreasing score for the test. From table 2, the post-test score elicited a statistically significant median increase compared to the pre-test score, in both modules, with a p-value < 0.000 (Table 2).
There was a weak positive correlation between the test score and the students' performance in the online tutorial, with p-value > 0.05 (Table 3). Table 3: Correlation between student performance with total test scores

Discussion
PBLs, consisting of small-group works, is student-centered, independent, and self-directed. This provides a motivating and enjoyable approach to medical education (Neufeld & Barrows, 1974). From this study, student performances were overall in the "outstanding" category. They showed good participation and communication skills, even though the discussions were done online via teleconferencing. They still showed the ability to apply and link concepts to the problem and draw valid conclusions (Table 1). Another study reported that students' performance in finding and evaluating evidence, organizing and prioritizing hypotheses, and making logical inferences during the online PBL was positive (Murata et al., 2021). Many studies had reported the use of PBL in online learning environments. Students use online discussions for supporting their argument. Due to its flexibility with time, online PBL makes students involved into more intense discussions. This study revealed that students meaningfully transferred what they had learned from online discussions to their actual discussions during PBL (Cho & Jonassen, 2002). Another example of online PBL is the one using WeChat in a dental clerkship which gives some advantages: a time-saving and convenient platform for discussion among students and tutors. This flexibility became a crucial part that allows asynchronous discussion anytime and anywhere (Zhang et al., 2019).
A study also indicated that online PBL could support access to information and collaborative learning and keep interactions among students (McLinden et al., 2006). A study with Massive Open Online Course (MOOC) where virtual teams worked on PBL-like tasks shows that they can collaborate on learning tasks. Additional support is needed to prepare participants for virtual teamwork, develop digital literacy, and stimulate more elaborate brainstorming and discussion (Verstegen et al., 2018). Another study concluded that the quality of online PBL was similar as

Variable
Total Score of Student Performance r-value p-value* Average total test score (in both modules) 0.145 0.168 compared to in-person/ face to face discussion (Nagge et al., 2018). A successful online PBL was dependent on the interactions, equal participation, and inclusion of all group members (Saqr et al., 2020).
To see whether the content knowledge was achieved, the post-test the score of the online PBL increased more significantly than the pre-test for both modules (Table 2). A previous study reported that learning in the online PBL group significantly increased students' critical thinking skills (Şendağ & Ferhan Odabaşı, 2009). This ability has a significant implication in the discussion; it has a crucial role in activating problem solving and decisions making process (Chaffee, 1994). Analysis and synthesis of information were reported as being enhanced during an online tutorial (Valaitis et al., 2005). A weak correlation between students' performance and test scores does not indicate that the online tutorial process was flawed (Table 3). Many factors can determine success in online courses. Self-regulation and motivation have been identified as the students' ability to plan, monitor, and evaluate their behaviour, cognition, and learning strategies (Matuga, 2009). Main platform for online PBL in FKIK Unismuh is Zoom. This application is quite effective for lecturer and students seeing each other, but internet connection problems become an obstacle. It needs a stable and strong network connection so the lecture can run smoothly using Zoom. Internet access that is not adequate in the village areas hinder the students to participate in online learning. There are still some areas which internet access has not been adequate (Lestiyanawati & Widyantoro, 2020). So, in designing online learning environments to support PBL, we need to provide students with infrastructure and communication facilities to work collaboratively.

Conclusion
There is a weak association between students' performance and their content of knowledge about the module shown by the increase of the test score. These limitations or obstacles during online PBL can be overcome with better preparation from the students with sufficient digital literacy, selfregulation, and adequate internet network support.