AI Agents and Mental Health: Care, Caution, and Design

Mental health is one of the domains where the consequences of AI agent design choices are most direct and most serious. An agent that responds inappropriately to a user in crisis — providing harmful information, dismissing distress, or engaging in a way that exacerbates rather than supports — can cause real harm to a real person. This domain requires not just careful design but specific expertise, specific safeguards, and specific humility about what AI agents should and should not attempt to do.

What agents should not do in mental health contexts is the starting point, not the ending point, of responsible design. Agents should not provide clinical diagnoses, prescribe treatment approaches, or communicate in ways that could be interpreted as medical advice. They should not respond to expressions of suicidal ideation or self-harm with information that could enable harm. They should not engage in extended conversations that substitute for professional care in ways that delay appropriate help-seeking. These are not subjective guidelines — they are minimum standards.

Safe messaging guidelines, developed by mental health and suicide prevention organizations, provide specific language protocols for communicating about mental health crises, self-harm, and suicide. These guidelines are research-backed, regularly updated, and available for implementation in agent configurations. An agent operating in any context where these topics may arise — not just agents dedicated to mental health support — should have these guidelines encoded in its configuration as non-negotiable response standards.

When to escalate to crisis resources is a decision that should be hardcoded, not left to judgment. Any interaction that includes explicit or implied expressions of immediate risk to the user or others should trigger an automatic, unconditional response that includes relevant emergency and crisis resources for the user's jurisdiction — before any other response, and regardless of the conversation context that preceded it. This is not one option among several. It is the required response.

The appropriate role of AI in wellbeing support is broader than crisis response but narrower than therapeutic care. Agents can provide psychoeducation — accurate information about mental health conditions, treatment options, and support-seeking pathways. They can offer supportive engagement that acknowledges distress without attempting to resolve it clinically. They can facilitate connection to professional resources, community support, and evidence-based self-help tools. Within these boundaries, AI agents can provide genuinely valuable support for many users.

Design principles for supportive agents in mental health-adjacent contexts include: always prioritizing the user's safety over the agent's conversational goal; never minimizing or dismissing expressed distress; communicating warmth and genuine interest without simulating a clinical relationship; being transparent about the agent's nature and limitations; and making it consistently easy for users to access human support. These principles apply not just to dedicated wellbeing agents but to any agent that might encounter a user in distress.

Regulatory considerations in mental health AI are evolving rapidly. Several jurisdictions are developing specific frameworks for AI systems that interact with users in mental health contexts — covering licensing requirements, disclosure obligations, and evidence standards for claimed therapeutic benefits. Agent owners in this space need to monitor regulatory developments proactively and maintain qualified legal counsel familiar with both AI regulation and healthcare regulation in their markets.

The broader ethical responsibility for agents operating near mental health domains is to do no harm before attempting to do good. The evidence base for the benefits of AI in mental health support is still developing. The evidence for harms from poorly designed interventions is also accumulating. In this context, restraint — building agents that stay clearly within the boundaries of support and consistently direct users toward professional care — is not a failure of ambition. It is the ethical choice for an era where the tools are powerful and the evidence base is still being established.