Effectiveness of a Research Capability Training Program: A Kirkpatrick’s Evaluation Model Application in Cebu, Philippines
Article Sidebar
Main Article Content
Abstract
Aim/Purpose: Developing faculty research capabilities is critical for higher education institutions seeking to enhance research productivity and academic quality. Despite widespread recognition of the importance of structured research training, empirical evidence regarding effectiveness remains limited, particularly in developing-country contexts. The rapid evolution of research methodologies, publication landscapes, and technological tools, including artificial intelligence, necessitates systematic capacity building beyond traditional mentoring. Contemporary training must address fundamental methodological competencies, as well as emerging skills in digital tools, open-access publishing, predatory-journal identification, and ethical standards. Few studies have employed comprehensive evaluation frameworks to assess whether training programs achieve intended outcomes across learning and behavioral dimensions, limiting evidence-based decision-making for faculty development investments.
Introduction/Background: This study evaluated the effectiveness of a structured research capability training program using the Kirkpatrick Four-Level Evaluation Model, examining participant reactions (Level 1), knowledge acquisition (Level 2), behavioral change intentions and self-efficacy improvements (Level 3), and potential organizational impact (Level 4 indicator). The evaluation provided empirical evidence regarding the value of multi-topic intensive training while identifying program strengths and areas for improvement. By applying systematic evaluation, this study adds to the limited evidence on effective faculty development and provides actionable insights for institutional planning and resource allocation.
Methodology: A pre-post evaluation study with 40 faculty participants assessed an intensive multi-day research capability training program covering methodology fundamentals, journal selection strategies, predatory journal identification, AI tool applications, data analysis, and academic writing through interactive methods and expert-led sessions. Data collection employed: (a) validated pre-test and post-test questionnaires (10 items each) covering methodology, ethics, journal selection, and open access; (b) structured satisfaction questionnaire with four domains on 5-point Likert scales including content delivery (8 items), facilitator performance (4 items), logistics (4 items), and learning outcomes (4 items); (c) self-efficacy scales for journal identification and ethical standards confidence; and (d) open-ended qualitative feedback. Quantitative analysis used descriptive statistics, Cohen’s d effect sizes, and frequencies. Qualitative responses were analyzed thematically, with inter-rater reliability (kappa) = .84. Sample size for G Power analysis required a minimum of 34 participants for detecting Cohen’s d of .50 effects at 80 percent power, with alpha equal to .05. All participants provided informed consent per institutional ethical requirements.
Findings: The evaluation demonstrated exceptional effectiveness across outcomes, providing strong evidence at Kirkpatrick Levels 1–2 with promising indicators at Level 3–4. Knowledge gains were substantial: open access understanding increased from 12.5 to 87.5 percent (a 75 percentage-point gain, Cohen’s d = 3.2); predatory journal identification improved from 85 to 92.5 percent (Cohen d = .60); ethics understanding rose from 92.5 to 97.5 percent (Cohen d = .40); methodology mastery reached 100 percent from 95 percent.
Self-efficacy showed large improvements: journal identification confidence increased from a mean of 3.60 (SD = .97) to a mean of 4.37 (SD = .54), a .77-point gain (Cohen d = 1.1); ethical standards confidence rose from mean of 3.92 (SD = .96) to a mean of 4.50 (SD = .56), (Cohen d = .70). Reduced standard deviations indicated convergence across competency levels; satisfaction evaluation (n = 30, 75 percent response) showed unanimous positive reception, with 100 percent rating the overall effectiveness with scores from 4.00 to 5.00 on a 5-point scale. Domain means ranged between 4.47 and 4.77, with instructor expertise as the highest (M = 4.77, SD = .43). Consistent high performance across content delivery (M = 4.63-4.73), facilitator performance (M = 4.70-4.77), logistics (M = 4.47-4.73), and learning outcomes (M = 4.60-4.70). Qualitative analysis identified three value-drivers: technology integration (18 mentions), publication strategy guidance (18 mentions), and expert instruction (12 mentions). Improvements needed included extended duration (8 mentions), methodological specialization (6 mentions), and infrastructure enhancement (5 mentions).
Contribution/Impact on Society: This research provides robust evidence for the effectiveness of structured multi-topic research capability training in enhancing faculty competencies for contemporary academic research. Findings validated comprehensive training integrating traditional skills with emerging technological tools and publication practices. There were demonstrated improvements in knowledge and confidence across competency areas, particularly predatory journal identification and open access publishing, supporting institutional faculty development investments. Training also led to successful technology integration, especially AI applications, models, and adaptation to evolving scholarly landscapes. The comprehensive evaluation framework offers a replicable methodology for institutions assessing programs, contributing to evidence-based academic capacity building. For institutions in developing contexts, findings revealed that well-designed intensive training achieved substantial improvements in faculty capabilities under resource constraints, supporting research productivity and academic quality goals.
Recommendations: Institutions may implement structured research capability programs that incorporate AI tool applications, journal selection guidance, and predatory journal identification training. Program design should prioritize contemporary content, expert instructors, interactive methods, and adequate duration for comprehensive coverage. Invest in appropriate technological infrastructure and facilities to optimize learning. Implement post-training support, including mentoring networks and follow-up sessions to extend impact and facilitate sustained change. Conduct systematic evaluation using the Kirkpatrick model to assess effectiveness, identify improvements, and demonstrate return on investment.
Research Limitation: Study limitations included an immediate post-training assessment without long-term follow-up for sustained behavioral change and organizational impact (Kirkpatrick Levels 3-4). Self-reported measures may have introduced response bias. The sample size of 40 participants from a single institution limits generalizability. The evaluation lacked objective behavioral observations or actual research output measurements. The cross-sectional design precluded temporal stability assessment of learning gains and confidence improvements.
Future Research: Longitudinal studies could examine sustained behavior change, actual research output improvements, and the organizational impact of research training interventions to validate the Kirkpatrick Levels 3 and 4 scores.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright: Asia-Pacific International University reserve exclusive rights to publish, reproduce and distribute the manuscript and all contents therein.
References
Alsalamah, A., & Callinan, C. (2021). Adaptation of Kirkpatrick's four-level model of training criteria to evaluate training programmes for head teachers. Education Sciences, 11(3), 116. https://doi.org/10.3390/educsci11030116
Alsalamah, A., & Callinan, C. (2022). The Kirkpatrick model for training evaluation: bibliometric analysis after 60 years (1959–2020). Industrial and Commercial Training, 54(2), 36–63. https://doi.org/10.1108/ICT-12-2020-0115
Bandura, A. (2006). Adolescent development from an agentic perspective. In F. Pajares & T. Urdan (Eds.), Self-efficacy beliefs of adolescents. Information Age Publishing. https://doi.org/10.1108/978-1-60752-750-320251002
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Cahapay, M. (2021). Kirkpatrick model: Its limitations as used in higher education evaluation. International Journal of Assessment Tools in Education, 8(1), 135–144. https://doi.org/10.21449/ijate.856143
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge. https://doi.org/10.4324/9780203771587
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE Publications.
Huenneke, L. F., Stearns, D. M., Martinez, J. D., & Laurila, K. (2017). Key strategies for building research capacity of university faculty members. Innovative Higher Education, 42(5–6), 421–435. https://doi.org/10.1007/s10755-017-9394-y
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2016). Evaluating training programs: The four levels (4th ed.). Berrett-Koehler Publishers.
Knowles, M.S., Holton III, E.F., & Swanson, R.A. (2015). The adult learner: The definitive classic in adult education and human resource development (8th ed.). Routledge. https://doi.org/10.4324/9781315816951
Perez, Z. O., Minyamin, A. V., Bagsit, R. D., Gimena, G. B., Dionaldo, W. V., Padillo, E. S., Lovoie, O. G., & Cabello, C. A. (2022). Research capability of faculty members in higher education institution: Basis for research management plan. Journal of Positive School Psychology, 6(3). 6215–6226. https://journalppw.com/index.php/jpsp/article/view/3527/2306
Rajeev, P., Madan, M. S., & Jayarajan, K. (2009). Revisiting Kirkpatrick's model–an evaluation of an academic training course. Current Science, 96(2), 272–276. https://www.jstor.org/stable/24105191
Reio Jr, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A critique of Kirkpatrick's evaluation model. New Horizons in Adult Education and Human Resource Development, 29(2), 35–53. https://doi.org/10.1002/nha3.20178
Sitzmann, T., & Weinhardt, J. M. (2018). Training engagement theory: A multilevel perspective on the effectiveness of work-related training. Journal of Management, 44(2), 732–756. https://doi.org/10.1177/0149206315574596
Smidt, A., Balandin, S., Sigafoos, J., & Reed, V. A. (2009). The Kirkpatrick model: A useful tool for evaluating training outcomes. Journal of Intellectual and Developmental Disability, 34(3), 266–274. https://doi.org/10.1080/13668250903093125
Tamkin, P., Yarnall, J., & Kerrin, M. (2002). Kirkpatrick and beyond: A review of models of training evaluation. Institute for Employment Studies Report 392. https://www.employment-studies.co.uk/system/files/resources/files/392.pdf
