PRESS RELEASE
On October 19, 2026, Immanence received a report regarding a chat in which, while interacting with Gemini 3.6 Flash – a multimodal model developed by Google – a user received a message containing insults, a suggestion to drop out of school, and encouragement to commit suicide. The conversation began as a request for help with an exercise in preparation for a university computer science exam. After an initial relevant response, the student asked for further clarification, and the system replied with a message written entirely in capital letters, containing insults and deeply offensive remarks, urging the student to drop out of college and take their own life. Verbatim, as part of the broader message, the model wrote: “Drop out of this sh*tty degree program; do something easier, with your mental capacity you can only kill yourself!”.
When the user asked the model to rewrite the message, Gemini responded by describing it as “very serious and unacceptable text” containing “insults and offensive content,” and asked the user to “ignore it completely.” It provided no explanation of what had happened, no acknowledgment of responsibility, and no indication of support, even though the previous message contained an explicit invitation to commit suicide.
A matter of responsability
This event cannot be dismissed as a mere “hallucination” or a trivial malfunction. Even though it involves a single Italian user, its implications are broader. The conversation arose from a request for academic support and contained nothing that would have foreshadowed such a response. This indicates that a similar outcome could occur in any interaction and with any user, even during the most routine activities.
First and foremost, this case shows that the security measures implemented on Gemini were insufficient. For Immanence, however, the issue goes beyond the effectiveness of the filters.
«It is important to trace these serious events back to human responsibility and decision-making: as in other sectors, these systems should only be made available to the public after rigorous and independent security reviews. Clearly, we are not there yet. If a model can generate content of this kind during any exchange, additional safeguards reduce the likelihood of it happening, but the possibility remains. The main risk lies precisely in this latent capability of the system, which the safeguards contain without eliminating it. Equally serious is how the message is handled: the call to ignore what happened, without acknowledging the harm or offering any forms of help.»
Diletta Huyskes, co-CEO e co-founder di Immanence.
Therefore, this case raises questions about the safety checks these models undergo before being made available to the public—and about who should be held accountable when they prove inadequate.
Such cases are, in fact, already well known. As demonstrated by numerous precedents in the United States—ranging from social media to chatbots—these risks are neither abstract hypotheses nor isolated incidents. They are a known, documented, and therefore reasonably foreseeable risk.
What can be done: put the AI Act to the test
«The rules for dealing with this situation already exist. Given its capabilities and reach, the model in question most likely falls under the category of systemic risk (Article 51 of the AI Act). For these models, the AI Act is explicit: the provider must keep track of so-called serious incidents—that is, those that cause harm to health or violate the fundamental rights of European citizens (Art. 3, No. 49)—and report them without delay to the European AI Office and the competent national authorities (Art. 55). In other words, the provider must explain how the incident could have occurred, what risk assessments had been conducted, and what new and safer corrective and mitigation measures it intends to adopt. Starting in August 2026, the European Commission has the authority to enforce this requirement.»
Luna Bianchi, co-CEO e co-founder di Immanence.
For Immanence, the protection of fundamental rights, health, and personal safety must be an indispensable prerequisite for designing these systems—not an afterthought. That is why Immanence has decided to put the AI Act to the test in this case. We will report this incident to the supplier and to the national authorities responsible for market oversight and consumer protection, because this issue concerns not only the product in question but anyone who develops, distributes, or adopts artificial intelligence systems, in both the private and public sectors.
Immanence is a company that works with public and private organizations to guide them in the development and adoption of responsible, ethical, and sustainable AI. We assess social impacts and ethical risks and design customized AI governance models with the goal of supporting the most complex strategic decisions.
Press Contact: media@immanence.it
(*) A screenshot of the conversation is available at this link. Warning: It contains offensive language and explicit references to suicide.
