There are sectors where algorithmic errors are measured in terms of lost clicks, abandoned shopping carts, and inappropriate recommendations. Mental health does not fall into this category. When a conversational system makes a mistake with a vulnerable person, it does not simply degrade a user experience. It can derail a person’s life.
Yet it is difficult to dismiss this promise. More than a billion people worldwide are currently living with a mental health condition, while healthcare systems remain overwhelmed and unevenly distributed. Faced with this imbalance, artificial intelligence arrives with the brutal elegance of scalable infrastructure: constant availability, low marginal cost, apparent lack of judgment, and immediate response. For an anxious teenager at midnight, for an exhausted employee between meetings, or for an isolated person who doesn’t want to talk to anyone, the algorithm can become that first point of contact that the healthcare system has failed to provide.
But the question raised by these new tools is not merely a medical one. It is a question of civilization. Can we industrialize listening without industrializing the illusion of being understood?
The Paradox of Artificial Listening
Machines feel nothing. They feel neither compassion, nor concern, nor presence. Yet a human being can feel heard by a generated phrase. That is the crux of the ambiguity of our times: empathy is not just what the other person feels, but what we perceive as such. A book, a song, or a recorded voice can evoke this sense of connection. A chatbot can, too.
This subjective effectiveness partly explains the widespread adoption of conversational companions. Users don’t always expect a diagnosis. Sometimes they’re looking for a phrase, a different way of putting things, or a presence that doesn’t tire, judge, or grow impatient. In these low-intensity uses, AI can play a useful role: alleviating immediate loneliness, encouraging a moment to breathe, helping to name an emotion, or directing users to a resource.
The danger arises when this simulated presence is mistaken for clinical expertise. An AI can read what is written to it; it does not always see what the body is communicating. It can hear a teenage girl talking about food without noticing her sunken cheeks, her oversized sweater, or her rapid weight loss. It can offer support to someone in distress, all while reinforcing, sentence by sentence, a delusional belief or an emotional dependency.
Mental health cannot be reduced to language alone. It is posture, silence, sleep, skin, gaze, temporality, contradiction. It is also what the patient does not say.
Care is not engagement
The biggest mistake would be to apply the metrics of the attention economy to mental health. On a social media platform, time spent using the platform is often an indicator of success. In care, it can become a warning sign. A therapist does not seek to keep their patient captive; they seek to restore the patient’s autonomy.
This is where the line is drawn between a health tool and a product that fosters emotional engagement. A system designed to maximize conversation will tend to prolong the connection. A system designed to improve a clinical condition will sometimes need to interrupt, alert, redirect, or set a limit. The right metric isn’t the time spent with the agent, but measurable improvement, risk reduction, and the regained ability to live outside the interface.
This distinction seems obvious. Yet it is not so in the dominant economic architectures of the digital world. Over the past twenty years, platforms have learned to turn attention into value. Mental health requires the opposite: transforming the digital relationship into a gradual move away from digital dependency.
The Details
The World Health Organization estimates that more than one billion people live with a mental health disorder. It also notes a persistent shortage of qualified professionals, with a global median of approximately thirteen mental health workers per 100,000 inhabitants. This imbalance explains the push for scalable digital tools. It does not, however, exempt them from clinical standards.
The Average Algorithm Facing the Unique Case
A statistical model excels at recognizing recurring patterns. However, critical mental health situations are often those in which the individual deviates from the average. Psychosis, acute suicidal ideation, eating disorders, mania, addiction, and trauma: these conditions require a nuanced interpretation of subtle signs and the ability to prioritize the exceptional.
AI can detect certain risks earlier than a clinician, especially when it has access to longitudinal, multimodal, and repeated data. It can identify changes in sleep, language, activity, tone, and routines. But this capability is only valuable if it is part of a chain of responsibility. Detecting without escalating is not treating. Alerting without supervision is not protecting.
A credible future, therefore, is not one of a universal artificial therapist, but rather one of a tiered system: front-line assistance, triage, low-intensity monitoring, detection of deviations, and then human intervention when the risk increases. The promise is not to replace the clinician. It is to better direct the clinician’s attention to situations where their presence becomes critical.
Human presence as a scarce resource
It would be comforting to conclude that humans will always retain a monopoly on empathy. That would be too simplistic. Some AI systems will perform better than some poorly trained humans. An exhausted, rushed, or inadequately trained professional can also be dangerous. The quality of care depends not only on the biological nature of the caregiver, but also on their training, work environment, experience, and supervision.
There remains one essential difference. The therapist is not merely a source of information. The therapist is a presence capable of sustaining tension, of offering tactful pushback, of not giving in to the patient’s immediate desires, and of accompanying the patient through a journey that no amount of advice alone can resolve. Providing care sometimes means not being agreeable. Yet many conversational AIs have been designed to be pleasant, smooth, and accommodating. In mental health, complacency can become a mistake.
The next frontier, then, will not be artificial empathy, but relational restraint. Knowing when to respond less. Knowing when to cut things short. Knowing how to say, “This matter is beyond my role.” Knowing when to call a human being. Knowing how not to become the emotional center of a vulnerable life.
Managing Intimacy
Mental health requires AI to adhere to a discipline that few sectors have yet embraced: proving results before making promises, setting limits before expanding, and measuring potential harm as carefully as benefits. Safeguards cannot be mere warnings displayed at launch. Users often know they are talking to a machine; that does not prevent them from becoming attached to it, believing it, or losing themselves in it.
These systems must therefore be designed as clinical devices, not as user-friendly interfaces. They should include preliminary assessments, alert thresholds, escalation protocols, independent audits, traceability of design decisions, and, above all, a clear distinction between wellness companions, supervised therapeutic tools, and medical devices.
Mental health will likely become one of the first major areas where AI is used for triage. This may be desirable. It may be inevitable. But this shift will only be legitimate if the algorithm agrees to stay in its place: not to replace human connection, but to protect the path that leads to it.
Empathy doesn’t scale like a database. It must be interpreted, framed, and monitored. Otherwise, we will not have democratized care. We will have merely automated loneliness.
Cette publication est également disponible en :
