Anthropic’s recent research suggests users grow complacent when Claude delivers highly structured outputs such as code or documents. These “artifacts” impress but users rarely double-check facts or context, even though complex tasks often hide errors.
The study sampled roughly 10,000 conversations over a week. In 86 per cent of cases, users revised their prompts for better results; just 30 per cent clarified response preferences up front. Anthropic urges users to scrutinise artifacts more closely and challenge assumptions, instead of accepting them at face value.
This phenomenon is cropping up for a few reasons, says Anthropic. Maybe Claude spits out responses so slick, users take them at face value; if the product looks ready for launch, why dig deeper?
Or perhaps artifact chats trend toward projects where polish trumps precision, like crafting a user interface instead of penning contracts. And who knows what feedback goes down off-platform: code put through its paces in another IDE, apps road-tested outside the chat, drafts passed around in private Slack channels, all without a word of critique posted back.
Whatever’s fuelling this shift, it’s worth a closer look.
As AI systems churn out ever more polished work, real value will come from those who can slice through the gloss and give smart, incisive feedback, whether in public threads or behind closed doors.
Steps to developing AI fluency:
Maintain engagement in the conversation. Iteration and refinement are strongly correlated with overall fluency behaviours observed in the research analysis. Consider your initial response as a foundation; seek clarification through follow-up questions, address areas of uncertainty, like pushing back on parts that feel odd, and continually refine your objectives.
Critically assess polished outputs. Even when AI-generated responses appear comprehensive or well presented, it is essential to pause and evaluate their accuracy and completeness. Verify whether the reasoning is sound and ensure that no critical details are overlooked. Anthropic’s research data indicates that users tend to perform less critical evaluation when presented with polished outputs, despite providing detailed instructions at the outset.
Define collaboration parameters. Only 30 per cent of users specify their preferred interaction style for Claude during conversations. For more effective collaboration, provide explicit guidance such as “Please challenge my assumptions if necessary”, “Explain your reasoning before offering a conclusion” or “Highlight any areas of uncertainty”. Clearly communicating these expectations at the beginning can significantly influence the course of the interaction.
But first, a caveat. Anthropic’s snapshot comes from Claude.ai users hashing out multi-turn chats during a single week in January 2026. With AI tools still on the bleeding edge, these participants skew towards early adopters: tech-savvy folks who already have some skin in the game. That means this data isn’t a catch-all for how everyone interacts with AI.
Instead, we ought to treat this as a baseline for a niche crowd rather than a gold standard. The one-week collection window leaves out seasonal shifts and evolving trends. And since the analysis zeroes in on Claude.ai, Anthropic advises that we shouldn’t expect insights about how users play with other AI platforms.
This is key to interrogating new ways of working with AI tools.
For instance, in preparation for this study, Anthropic conducted some initial analysis that found consistency between Claude Code conversations and ones in Claude.ai. But this is still preliminary, and Claude Code’s very different user base and functionality implies that more substantial research is necessary.
Experimenting with fresh ways to harness AI tools is fast becoming mission-critical. Anthropic’s researchers are dialling up their ambitions. They’re planning “cohort analyses”, meaning that they’ll track how newbies stack up against power users as they level up their AI fluency.
The team’s also breaking out qualitative research to dive into quirky user habits that don’t show up in vanilla Claude.ai chat logs. And they want answers to the big stuff: Does nudging users toward back-and-forth dialogues actually spark sharper critical thinking? Or are there smarter hacks?
Watch this space.
