Bias-Aware Content Validity Assessment: Expert Judgment, Power Asymmetry, and Validity Evidence in Research Instrument Development
Article Sidebar
Main Article Content
Abstract
This academic article aims to propose a structured framework for evaluating item quality that accounts for evaluative distortions by examining how hierarchical relationships and subjective dynamics impact measurement integrity during tool construction. This article employs conceptual literature synthesis based on 40 academic and research sources selected from scholarly databases or journal indexes, together with foundational and widely cited works in relevant fields. The synthesis covers content validity, research instrument development, validity evidence, expert judgment, assessment bias, power asymmetry, consensus processes, the Delphi technique, and structured expert elicitation.
The synthesis indicates that content validity assessment should not rely solely on IOC, I-CVI, or S-CVI/Ave. These indices should be considered together with the quality of the assessment process, including content-domain specification, expert selection, independent rating, justification of scores, management of divergent comments, and documentation of item revisions. This article argues that expert consensus should be interpreted with caution because it may reflect familiarity with existing frameworks, judgment bias, or hidden power dynamics rather than the substantive quality of the items themselves. The main contribution of this article is the Bias-Aware Content Validity Framework, which helps researchers design a more transparent, auditable, and academically rigorous process for validating research instruments.
Article Details
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
Hatyai Academic Journal is an open access journal licensed under Creative Commons Attribution-NoDerivatives (CC BY-NC-ND 4.0). Readers are free to access, download, and share the content for educational purposes at no cost, provided that the work is not modified and proper attribution is given.
Authors retain the copyright of their work and grant the journal right of first publication.
References
Ahearn, E.-R., Ablaza, C., Stambe, R.-M., & Parsell, C. (2025). The clarity ledger for evaluation and applied research reporting (CLEAR-R). Research Evaluation, 34, rvaf046. https://doi.org/10.1093/reseval/rvaf046
Almanasreh, E., Moles, R. J., & Chen, T. F. (2022). A practical approach to the assessment and quantification of content validity. In Contemporary research methods in pharmacy and health services (pp. 583–599). https://doi.org/10.1016/B978-0-323-91888-6.00013-2
Avella, J. R. (2016). Delphi panels: Research design, procedures, advantages, and challenges. International Journal of Doctoral Studies, 11, 305–321. https://doi.org/10.28945/3561
Ay, T., & Özdemir, A. (2025). Perception of Cognitive Biases in Decision-Making Scale (PCBDM-S): Development and initial validation of a self-report measure for assessing cognitive bias perception. Current Psychology, 44(12), 12820–12834. https://doi.org/10.1007/s12144-025-08053-x
Barnaud, C., & Paassen, A. van. (2013). Equity, power games, and legitimacy: Dilemmas of participatory natural resource management. Ecology and Society, 18(2), 21. https://doi.org/10.5751/ES-05459-180221
Beck, K. (2020). Ensuring content validity of psychological and educational tests: The role of experts. Frontline Learning Research, 8(6), 1–37. https://doi.org/10.14786/flr.v8i6.517
Belzak, W. C. M., Naismith, B., & Burstein, J. (2023). Ensuring fairness of human- and AI-generated test items. Communications in Computer and Information Science, 1831, 701–707. https://doi.org/10.1007/978-3-031-36336-8_108
Bradley, P. H. (1978). Power, status, and upward communication in small decision-making groups. Communication Monographs, 45(1), 33–43. https://doi.org/10.1080/03637757809375949
Brussel, S. van., Timmermans, M., Verkoeijen, P., & Paas, F. (2020). “Consider the opposite”: Effects of elaborative feedback and correct answer feedback on reducing confirmation bias—A pre-registered study. Contemporary Educational Psychology, 60, 101844. https://doi.org/10.1016/j.cedpsych.2020.101844
Brussel, S. van., Timmermans, M., Verkoeijen, P., & Paas, F. (2021). Teaching on video as an instructional strategy to reduce confirmation bias: A pre-registered study. Instructional Science, 49(4), 475–496. https://doi.org/10.1007/s11251-021-09547-4
Chen, N., Liu, J., Dong, X., Liu, Q., Sakai, T., & Wu, X.-M. (2024). AI can be cognitively biased: An exploratory study on threshold priming in LLM-based batch relevance assessment. In SIGIR-AP 2024: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (pp. 54–63). Tokyo, Japan. https://doi.org/10.1145/3673791.3698420
Delgado-Rico, E., Carretero-Dios, H., & Ruch, W. (2012). Content validity evidences in test development: An applied perspective. International Journal of Clinical and Health Psychology, 12(3), 449–460.
Ercikan, K., Arim, R., Law, D., Domene, J., Gagnon, F., & Lacroix, S. (2010). Application of think aloud protocols for examining and confirming sources of differential item functioning identified by expert reviews. Educational Measurement: Issues and Practice, 29(2), 24–35. https://doi.org/10.1111/j.1745-3992.2010.00173.x
Evans, B. C. (2005). Content validation of instruments: Are the perspectives of Anglo reviewers different from those of Hispanic/Latino and American Indian reviewers?. Journal of Nursing Education, 44(5), 216–224. https://doi.org/10.3928/01484834-20050501-04
Gorst, S. L., Prinsen, C. A. C., Salcher-Konrad, M., Matvienko-Sikar, K., Williamson, P. R., & Terwee, C. B. (2020). Methods used in the selection of instruments for outcomes included in core outcome sets have improved since the publication of the COSMIN/COMET guideline. Journal of Clinical Epidemiology, 125, 64–75. https://doi.org/10.1016/j.jclinepi.2020.05.021
Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Hawkins, M., Elsworth, G. R., Hoban, E., & Osborne, R. H. (2020). Questionnaire validation practice within a theoretical framework: A systematic descriptive literature review of health literacy assessments. BMJ Open, 10(6), e035974. https://doi.org/10.1136/bmjopen-2019-035974
Haynes, S. N., Matsui, M., & Kaholokula, J. K. (2026). Content validation and cultural adaptation are necessary to interpret evidence from psychological assessment instruments imported across cultures. Current Opinion in Psychology, 70, 102307. https://doi.org/10.1016/j.copsyc.2026.102307
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238–247. https://doi.org/10.1037/1040-3590.7.3.238
Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F., & Wintle, B. C. (2018). A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 9(1), 169–181. https://doi.org/10.1111/2041-210X.12857
Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755
Li, S.-A., Yousefi-Nooraie, R., Guyatt, G., Talwar, G., Wang, Q., Zhu, Y., Hozo, I., & Djulbegovic, B. (2022). A few panel members dominated guideline development meeting discussions: Social network analysis. Journal of Clinical Epidemiology, 141, 1–10. https://doi.org/10.1016/j.jclinepi.2021.09.023
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017
Mokkink, L. B., Prinsen, C. A. C., Bouter, L. M., Vet, H. C. W. de., & Terwee, C. B. (2016). The consensus-based standards for the selection of health measurement instruments (COSMIN) and how to select an outcome measurement instrument. Brazilian Journal of Physical Therapy, 20(2), 105–113. https://doi.org/10.1590/bjpt-rbf.2014.0143
Mumpower, J. L., & Stewart, T. R. (1996). Expert judgement and expert disagreement. Thinking & Reasoning, 2(2–3), 191–212. https://doi.org/10.1080/135467896394500
Nasa, P., Jain, R., & Juneja, D. (2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116–129. https://doi.org/10.5662/wjm.v11.i4.116
Newman, I., Lim, J., & Pineda, F. (2013). Content validity using a mixed methods approach: Its application and development through the use of a table of specifications methodology. Journal of Mixed Methods Research, 7(3), 243–260. https://doi.org/10.1177/1558689813476922
Paiva, S. M., Perazzo, M. F., Ortiz, F. R., Pordeus, I. A., & Martins-Júnior, P. A. (2018). How to select a questionnaire with a good methodological quality?. Brazilian Dental Journal, 29(1), 3–6. https://doi.org/10.1590/0103-6440201802008
Parker, M., Pearson, C., Donald, C., & Fisher, C. B. (2019). Beyond the Belmont Principles: A community-based approach to developing an Indigenous ethics model and curriculum for training health researchers working with American Indian and Alaska Native communities. American Journal of Community Psychology, 64(1–2), 9–20. https://doi.org/10.1002/ajcp.12360
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what is being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
Poll, G. H., & Petru, J. (2023). Exploring the contributions of expert review and cognitive interviewing to evaluating the content validity of items for a new measure of adolescent social communication, the transition pragmatics interview. Clinical Linguistics & Phonetics, 37(12), 1124–1140. https://doi.org/10.1080/02699206.2022.2148131
Saposnik, G., Redelmeier, D., Ruff, C. C., & Tobler, P. N. (2016). Cognitive biases associated with medical decisions: A systematic review. BMC Medical Informatics and Decision Making, 16, 138. https://doi.org/10.1186/s12911-016-0377-1
Sireci, S. G., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1), 100–107. https://doi.org/10.7334/psicothema2013.256
Smith, M. U. (1992). Expertise and the organization of knowledge: Unexpected differences among genetic counselors, faculty, and students on problem categorization tasks. Journal of Research in Science Teaching, 29(2), 179–205. https://doi.org/10.1002/tea.3660290207
Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104, 333–339. https://doi.org/10.1016/j.jbusres.2019.07.039
Svihla, V., & Gallup, A. (2021). Do loss aversion and the ownership effect bias content validation procedures?. Practical Assessment, Research & Evaluation, 26, 1–12. https://doi.org/10.7275/34d8-qe13
Terwee, C. B., Prinsen, C. A. C., Chiarotto, A., Westerman, M. J., Patrick, D. L., Alonso, J., Bouter, L. M., Vet, H. C. W. de., & Mokkink, L. B. (2018). COSMIN methodology for evaluating the content validity of patient-reported outcome measures: A Delphi study. Quality of Life Research, 27(5), 1159–1170. https://doi.org/10.1007/s11136-018-1829-0
Torraco, R. J. (2005). Writing integrative literature reviews: Guidelines and examples. Human Resource Development Review, 4(3), 356–367. https://doi.org/10.1177/1534484305278283
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124
Vickers, C. H. (2010). The local construction of the asymmetrical power relationship in teamwork among engineers. Critical Inquiry in Language Studies, 7(2–3), 131–151. https://doi.org/10.1080/15427581003757434
Wells-Di Gregorio, S., Deshields, T., Flowers, S. R., Taylor, N., Robbins, M. A., Johnson, R., Dwyer, M., Siston, A., Cooley, M. E., & Kayser, K. (2022). Development of a psychosocial oncology core curriculum for multidisciplinary education and training: Initial content validation using the modified Delphi method. Psycho-Oncology, 31(1), 130–138. https://doi.org/10.1002/pon.5791
