Computers in Human Behavior: Artificial Humans
2026
Shared — curated by The Gufo editorial team
User intent recognition and satisfaction with large language models: A user study with ChatGPT
Anna Bodonhelyi*, Efe Bozkir, Shuo Yang, Enkelejda Kasneci, Gjergji Kasneci
Received: 2026-10-22 · Published: 2026-12-01
© 2026 The Author(s) and the original publisher.
Abstract
The rapid evolution of LLMs represents an impactful paradigm shift in digital interaction and content engagement. While they encode vast amounts of human-generated knowledge and excel in processing diverse data types, they often face the challenge of accurately responding to specific user intents, leading to user dissatisfaction. Based on a fine-grained intent taxonomy and intent-based prompt reformulations, we analyze the quality of intent recognition and user satisfaction with answers from intent-based prompt reformulations of GPT-3.5 Turbo and GPT-4 Turbo models. Our study highlights the importance of human-AI interaction and underscores the need for interdisciplinary approaches to improve conversational AI systems. We show that GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents. Moreover, whenever the user intent is correctly recognized, while users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5, they tend to be more satisfied with the models' answers to their original prompts compared to the reformulated ones. The collected data from our study has been made publicly available on GitHub (https://github.com/ConcealedIDentity/UserIntentStudy) for further research.
Summary
This paper investigates how well large language models like GPT-3.5 Turbo and GPT-4 Turbo understand what users actually want when they type a prompt, and how satisfied users are with the responses they get. Using a fine-grained taxonomy of user intents, the researchers created reformulated versions of user prompts based on identified intents and compared the models' performance in recognizing these intents and the quality of their answers.
The study reveals a nuanced picture of model performance: GPT-4 is better at identifying frequently occurring intents, but GPT-3.5 sometimes does better with rarer, less common intents. Interestingly, even when intent recognition succeeds, users often prefer the model's response to their original, unmodified prompt over the version rewritten to explicitly match the recognized intent. The authors argue these findings highlight the complexity of human-AI interaction and the need for interdisciplinary approaches to improve conversational AI systems, and they have released their study data publicly for further research.
Plain-language summary by The Gufo; the paper is the authors’ own text.
Key findings
- GPT-4 Turbo outperforms GPT-3.5 Turbo at recognizing common user intents.
- GPT-3.5 Turbo often outperforms GPT-4 Turbo at recognizing less frequent user intents.
- When intent is correctly recognized, users are more satisfied with GPT-4's intent-based reformulated answers than with GPT-3.5's.
- Users tend to prefer the models' answers to their original prompts over answers to intent-based reformulated prompts.
