A recent study conducted by researchers from Brigham Young University (BYU), US, and 186 other universities compared ChatGPT, OpenAI’s chatbot product, to students on accounting exams.
Although students fared better overall, the researchers hailed ChatGPT’s performance as “impressive” and a “game changer” for education.
Performance Comparison: In the accounting exam, students achieved an average score of 76.7 percent, while ChatGPT scored 47.4 percent.
ChatGPT scored higher than the student average in 11.3 percent of questions, particularly excelling in accounting information systems (AIS) and auditing.
However, it struggled in tax, financial, and managerial assessments, possibly due to the mathematical processes involved.
Question Types: ChatGPT performed better on true/false questions (68.7 percent correct) and multiple-choice questions (59.5 percent), but short-answer questions proved more challenging, with scores ranging from 28.7 to 39.1 percent.
Higher-order questions were also more difficult for ChatGPT, as it sometimes provided incorrect explanations or different answers for the same question.
Accuracy and Fact-Checking:
The study found that ChatGPT sometimes provided explanations for incorrect answers and made up facts.
It also made nonsensical mathematical errors such as adding two numbers in a subtraction problem or dividing numbers incorrectly.
Recruitment and Methodology:
Lead study author David Wood recruited professors from 186 educational institutions in 14 countries to contribute 25,181 classroom accounting exam questions, while undergraduate BYU students provided another 2,268 textbook test bank questions to ChatGPT.
The questions covered AIS, auditing, financial accounting, managerial accounting and tax, and varied in difficulty and type.
Conclusion: The study’s authors noted that the results add to the ongoing debate about the role of AI models like ChatGPT in education.
While the bot’s performance was not perfect, it has the potential to revolutionize teaching and learning.
Future research will likely continue to explore the benefits and drawbacks of using AI in education.
Commentary:
The study sheds light on the capabilities and limitations of AI in education.
While ChatGPT’s performance was impressive, it’s not yet at a level where it can completely replace human instructors.
However, it could potentially be used as a valuable tool to supplement traditional teaching methods, providing personalized learning and immediate feedback to students.
It also highlights the importance of fact-checking and accuracy when using AI models for educational purposes.
Further research and development of AI in education will be an exciting area to watch in the coming years.