A user study on The Gufo's radar finds that GPT-4 reads common intents better than GPT-3.5, but reformulating prompts does not always please users more.
A new paper reports that GPT-4 Turbo recognises common user intents more accurately than GPT-3.5 Turbo, but GPT-3.5 is often better at recognising less frequent intents, according to the authors. Even when the system correctly identifies what a user wants, people are more satisfied with GPT-4's intent-based reformulations than with GPT-3.5's. Yet across both models, users tend to prefer the answers to their original prompts over answers to the reformulated versions.
The authors built a fine-grained taxonomy of user intents, then tested intent-based prompt reformulations against the two models in a study with real users who rated their satisfaction with the resulting answers.
The paper itself flags a tension in its own results: better intent recognition does not translate into better user satisfaction overall, since reformulated prompts underperform original ones in the users' judgement, a finding the authors say calls for more interdisciplinary work on human-AI interaction.
For practitioners building conversational systems, the lesson is that correctly labelling a user's intent and rewriting their prompt accordingly is not enough. User preference depends on more than semantic accuracy, and this dataset, released on GitHub, offers a concrete starting point for studying why.
The research
Anna Bodonhelyi, Efe Bozkir, Shuo Yang, Enkelejda Kasneci, Gjergji Kasneci (2026). User intent recognition and satisfaction with large language models: A user study with ChatGPT. Computers in Human Behavior Artificial Humans. https://doi.org/10.1016/j.chbah.2026.100400
Licence: CC BY (terms). The Gufo’s commentary is ours; the paper’s text stays with its authors.
