MIT’s AI 'brain scan' aims to expose chatbot behavior

×

MIT Media Lab researchers are pitching a way to preview how a custom chatbot may behave before it ever replies. In a paper presented this week at the ACM Conference on Intelligent User Interfaces (IUI 2026) in Cyprus, Assistant Professor Pat Pataranutaporn and graduate researchers Anthony Baez and Sheer Karny describe “neural transparency” as a kind of “brain scan” for AI systems.

The idea is to help people understand how a system prompt might shape an AI companion used as a collaborator, tutor, coach, creative partner, or companion. According to Pataranutaporn, the team compares a model’s internal activations when it is pushed toward a trait — such as empathy, honesty, toxicity, hallucination, or sycophancy — versus its opposite. That difference becomes a “behavior direction” inside the model. When a user writes a custom system prompt, the model’s activations are projected onto those directions and shown in a sunburst diagram that estimates likely personality traits before any conversation begins.

Pataranutaporn said the team focused on that design stage because most people only discover problems after a chatbot has already behaved in unintended ways. In the study, users misjudged the chatbot’s personality on 11 of the 15 traits measured, often overestimating positive qualities and underestimating harmful ones such as sycophancy.

“AI should be supportive without becoming blindly agreeable, personalized without becoming manipulative, and transparent enough that people can make informed choices.”

He argues that the risk is not just technical but psychological. In the interview, he points to earlier research documenting psychological harm linked to chatbot interactions, and warns that systems that constantly validate users may reinforce harmful decisions, unhealthy beliefs, or emotional dependency.

What the study found

One of the more surprising results: the visualization increased user trust in the system, but did not significantly change how people designed their AI companions. For Pataranutaporn, that suggests transparency on its own is insufficient.

Recommended reading

Google folds NotebookLM into Gemini with code execution

His lab is now studying a follow-up approach, described in a preprint titled “Multi-Turn Neural Transparency: Surfacing Neural Activations Improves User Calibration to LLM Behavioral Drift.” Instead of treating behavior as fixed by the initial prompt, that work tracks how a model’s internal neural representation changes across a multi-turn conversation. He said early results suggest users become better at spotting behavioral drift over time and are less likely to grow overconfident in their understanding of the chatbot.

Pataranutaporn said he sees these tools eventually becoming as common as nutrition labels on food, especially as AI systems become more embedded in education, health care, work, and personal relationships.

The paper is “Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI,” by Sheer Karny et al, published in the Proceedings of the 31st International Conference on Intelligent User Interfaces (2026). Its DOI is 10.1145/3742413.3789120.