Evaluating LLM-Generated Q&A Test: A Student-Centered Study

Anna Wróblewska; Bartosz Grabek; Jakub Świstak; Daniel Dan

Evaluating LLM-Generated Q&A Test: A Student-Centered Study

Anna Wróblewska
, Bartosz Grabek
, Jakub Świstak
, Daniel Dan

Technical University of Warsaw

Research output: Contribution to conference › Other › Research › peer-review

Abstract

This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o–based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star-ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.

Original language	English
Pages	277
Number of pages	289
Publication status	Published - 15 Jul 2025
Event	AIED 26th International Conference on Artificial intelligence in Education - University of Palermo, Palermo, Italy Duration: 22 Jul 2025 → 26 Jul 2025 Conference number: 26 https://aied2025.itd.cnr.it

Conference

Conference	AIED 26th International Conference on Artificial intelligence in Education
Abbreviated title	AIED
Country/Territory	Italy
City	Palermo
Period	22/07/2025 → 26/07/2025
Internet address	https://aied2025.itd.cnr.it

Access to Document

https://link.springer.com/chapter/10.1007/978-3-031-98417-4_20Licence: Unspecified

Cite this

@conference{355c238b82ec49eeb6b335fd083d8ac3,

title = "Evaluating LLM-Generated Q\&A Test: A Student-Centered Study",

abstract = "This research prepares an automatic pipeline for generating reliable question-answer (Q\&A) tests using AI chatbots. We automatically generated a GPT-4o–based Q\&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star-ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.",

author = "Anna Wr{\'o}blewska and Bartosz Grabek and Jakub {\'S}wistak and Daniel Dan",

year = "2025",

month = jul,

day = "15",

language = "English",

pages = "277",

note = "AIED 26th International Conference on Artificial intelligence in Education, AIED ; Conference date: 22-07-2025 Through 26-07-2025",

url = "https://aied2025.itd.cnr.it",

}

TY - CONF

T1 - Evaluating LLM-Generated Q&A Test: A Student-Centered Study

AU - Wróblewska, Anna

AU - Grabek, Bartosz

AU - Świstak, Jakub

AU - Dan, Daniel

N1 - Conference code: 26

PY - 2025/7/15

Y1 - 2025/7/15

N2 - This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o–based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star-ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.

AB - This research prepares an automatic pipeline for generating reliable question-answer (Q&A) tests using AI chatbots. We automatically generated a GPT-4o–based Q&A test for a Natural Language Processing course and evaluated its psychometric and perceived-quality metrics with students and experts. A mixed-format IRT analysis showed that the generated items exhibit strong discrimination and appropriate difficulty, while student and expert star-ratings reflect high overall quality. A uniform DIF check identified two items for review. These findings demonstrate that LLM-generated assessments can match human-authored tests in psychometric performance and user satisfaction, illustrating a scalable approach to AI-assisted assessment development.

M3 - Other

SP - 277

T2 - AIED 26th International Conference on Artificial intelligence in Education

Y2 - 22 July 2025 through 26 July 2025

ER -

Evaluating LLM-Generated Q&A Test: A Student-Centered Study

Abstract

Conference

Access to Document

Fingerprint

Cite this