Large Language Models (LLMs) like Gemini and ChatGPT are embedded into our work and daily routines. This is especially evident in software development, where AI transforms frontend and backend engineering processes. But as these systems take on more responsibility, one question looms large: how do we know if an LLM is truly helpful and trustworthy to its users?

Traditionally, AI models have been judged by their performance on academic benchmarks, but these tests often fail to capture what real users actually need. This gap has motivated user-centric evaluation approaches, which measure LLMs based on their ability to satisfy genuine user intents and preferences, rather than just scoring well on predefined tasks.

The Limitations of Traditional Benchmarks

Early LLM benchmarks were largely ability-focused. They treated LLMs as problem solvers for narrow tasks, measuring skills in isolation (e.g., knowledge recall, math, coding). Example: MMLU tests knowledge across many domains with exam-style questions, HumanEval checks if LLM models can write correct code, and many others. Each of these benchmarks targets a specific capability and scores the model’s accuracy or correctness in a controlled setting.

One major drawback of these traditional benchmarks is that they may not reflect tasks users truly care about. Furthermore, classic benchmarks tend to involve single-turn or single-step interactions (e.g., one question, one answer), whereas real user interactions often consist of multi-turn dialogues or multi-step problems. A user’s intent often spans multiple abilities such as creativity, knowledge, planning, and so on. Traditional benchmarks, organized by isolated skills, struggle to capture how well an LLM handles such complex, blended tasks.

User-First Approach to LLM Evaluation

User-centric evaluation closes key gaps by putting the user’s perspective at the center of how we benchmark LLMs. Instead of contrived tasks, it builds tests around real scenarios and intents.

For example, Wang et al. introduced User-Reported Scenarios (URS): 1,846 authentic interactions from 712 users across 23 countries, spanning everything from factual lookups to advice and creative brainstorming. Using vetted, firsthand data, models are evaluated on how well they serve as cooperative services that meet user needs (encompassing clarity, concision, relevance, and alignment with intent), shifting the focus from abstract capability to actual user satisfaction.

What sets these benchmarks apart? First, they are intent-driven. Rather than sorting by narrow skills (e.g., math vs. coding), they organize around what the user is trying to do — factual Q&A, professional problem-solving, creative ideation, advice-seeking, and more. Because real intents often require a blend of capabilities (retrieval, tailored reasoning, even empathy), intent-level evaluation shows precisely where a model meets expectations or falls short. For instance, giving concise, accurate answers to factual questions or offering relevant, thoughtful suggestions when asked for advice.

Second, they are multi-dimensional and inclusive. URS deliberately sampled queries from diverse cultures (across Asia, Europe, the Americas, and Africa) and in multiple languages (English and Chinese). This reduces bias toward English-only contexts and tests culture-specific knowledge, such as local calendars or idioms, that real users bring. Traditional benchmarks often skew toward English and a few domains, obscuring performance for non-English speakers and region-specific queries.

Finally, user-centric frameworks incorporate human feedback loops. Some collect explicit ratings from users or trained judges on correctness, clarity, and usefulness. In contrast, others analyze real queries from usage logs (with permission) to assess how models perform in practice. The goal is to approximate a simple question, “Would a real user be satisfied with this answer?” that closed-ended accuracy scores often miss. Studies show that user-centric scores correlate with self-reported satisfaction and reveal persistent weaknesses on subjective, open-ended tasks (opinions and advice), highlighting blind spots traditional tests can overlook.

Notable User-Centric Benchmarks and Frameworks

In response to the need for more trustworthy, user-grounded evaluations, researchers and organizations have developed several new benchmarks and frameworks. Below are a few prominent examples, each illustrating a different facet of the user-centric evaluation movement:

• Holistic Evaluation of Language Models (HELM): A comprehensive framework introduced by Stanford’s Center for Research on Foundation Models (CRFM). HELM is designed as a “living” benchmark covering a broad spectrum of real-world scenarios and metrics
• MT-Bench (Multi-Turn Benchmark): Developed by an academic collaboration (Zheng et al., 2024), MT-Bench specifically targets the dialogue experience with chat-based LLMs. It consists of 80 carefully written multi-turn conversation questions in English.
• AlpacaEval: An early user-focused benchmark (Li et al., 2023) that emerged from the Stanford Alpaca project. AlpacaEval assembled a set of 805 prompts covering diverse tasks, drawn from existing datasets and some model-generated (synthetic) queries.
• Real User Log Benchmarks (WildBench & AlignBench): These initiatives took an approach of mining real user interactions from specific platforms. WildBench (Lin et al., 2024) gathered 1,024 example questions from actual ChatGPT usage logs, and AlignBench (Liu et al., 2023) did similarly using logs from a Chinese LLM (ChatGLM).

Each of these examples contributes to the development of an evolving user-centric evaluation framework.

Conclusion and Outlook

Evaluating language models with real people in mind is a big step toward AI you can actually trust. Instead of chasing abstract test scores, this approach looks at what matters in everyday use: the variety of user goals, the back-and-forth of honest conversations, and whether people walk away satisfied. For teams choosing or building AI, that’s good news — progress gets measured by clarity, helpfulness, and fairness, not just by numbers on a leaderboard. Early user-focused studies are already surfacing gaps, particularly in open-ended, subjective tasks where human preferences are most significant.

Looking ahead, these benchmarks will get more nuanced. They’ll check whether a model follows instructions reliably and safely, and whether it protects privacy and security. Efforts like HELM are already tracking issues such as toxicity and bias to make sure models are not only useful but also respectful and fair. This goal is central to leading AI innovation in line with national action plans on responsible AI.

As LLMs roll out into more settings, we’ll see “living” benchmarks that update their scenarios and metrics to match new applications and shifting social expectations. That kind of dynamic evaluation will help redefine “state of the art” to mean not just cleverness in theory, but real value in practice.

User-centric evaluation reframes success around trust and satisfaction. It complements traditional tests by asking a deeper question — can this system actually meet my needs in a way I find trustworthy? Grounding assessments in real-world use and human feedback, we can confidently choose and integrate AI assistants into our work and daily lives.