Science Fiction and Sensationalism
Since we’ve invoked science fiction and sensationalism, it’s worth saying something about them. In both cases, the protagonist is a machine with human-like intelligence, or even superior intelligence, that at some point breaks free from human control and tends to either eliminate or subjugate the human species, or at least manipulate it. Science fiction stories offer valuable insights into the societies in which they were created, often serving as critiques of contemporary issues. In doing so, they present unique opportunities to reconsider the role of technology in our lives, economy, and politics. Sensationalism draws heavily from science fiction stories, not to critique contemporary reality, but to create distortions and alarmism about a hypothetical future we should really worry about. Examples are claims of long-termism, AGI, or Superintelligence, i.e., sentient artificial intelligence systems with their own intentions, opinions, and will, capable of manipulation if they wish. The point is that sensationalism serves only to divert our attention from current issues to purely hypothetical ones. Regarding manipulation, the issue is reduced to the possibility (or impossibility) of having machines that want to manipulate. But technological manipulation doesn’t necessarily have to resemble this science-fiction vision.
Manipulative Technologies: Cambridge Analytica, Deceptive Interfaces, Dark Patterns
First of all, there are cases of technological manipulation that depart from the science-fiction narrative, which have entered the public discourse in recent years.One of the most notorious examples is certainly Cambridge Analytica, the first major political manipulation scandal, in which millions of Facebook users’ data were exploited to run targeted campaigns promoting Brexit in the UK and supporting Trump in the 2016 US presidential election. The case shocked many and heightened public awareness about the influence that recommendation systems wield over users – and, by extension, about the power of platforms.
Other elements considered manipulative in the technological/digital context can be found in user interfaces. Dark patterns, a set of deceptive interface design strategies for the user, are the most obvious example of how digital interaction space can unknowingly lead the user to make disadvantageous choices. Common dark patterns manipulate users into consenting to share personal data, making impulsive purchases, or subscribing to unfavorable services. While not typically classified as a dark pattern, infinite scrolling can also be considered in this context. It effectively removes the user’s intentional choice to watch the next video, encouraging endless engagement. Dark patterns are interesting because they emphasize an important element of technological manipulation: the interaction context is skillfully designed around the cognitive and emotional capacities of users, and their perception of technology. Infinite scrolling exploits the human brain’s dopamine secretion mechanism, and by eliminating the factor of choice, it keeps the user on the platform, which is known as “doom-scrolling”.
All this to say that we don’t necessarily need a sentient technology to talk about manipulative technologies. Sometimes, even design choices that seem minimal can have a huge impact on decision-making processes. The online interaction and choice context is never a “neutral” place; it is always designed with pre-established interaction logics, and so users enter environments already built to facilitate certain behaviors. Clearly, not all of these behaviors disadvantage the user; in fact, most interfaces are designed to facilitate navigation, but that does not change the fact that they remain non-neutral or not deceitful. In fact, it’s essential to recognize that deception doesn’t always harm the user because, as mentioned, it is often functional to the consumption of the technology. The point is that deception is constitutive of digital environments. A useful concept to understand this interpretation of deception is “banal deception” (Natale, 2021), whereby interfaces are designed in such a way that the user actively falls into the illusion of the environment they interact with, much like a reader enters the illusion of reading.
That said, how do we go from manipulative technologies to manipulative chatbots?
Manipulative Chatbots
The shift from manipulation through deceptive interfaces and recommendation systems to chatbots is characterized by one aspect: interaction through conversation.
Chatbots are technologies with which we can interact as if we were speaking with a human being (and their strength lies in this). However, this opens up new forms of vulnerability. Let’s try to understand how chatbots can be manipulative. We can distinguish two categories: 1) elements in the interface (in terms of design choices), and 2) elements in the outputs that are the product of probabilistic processes.
Key elements for the user to fall into the illusion that the chatbot is intelligent are found in various language elements: the use of personal pronouns (I, me, my, etc.), the use of lexicon related to the mental sphere (I think, I believe, etc.), the use of lexicon related to the emotional sphere (I’m sorry, I’m happy, etc.), and the use of language itself, which humans recognize as an indicator of agency.
All of these elements can be considered intentional, as they are explainable through decisions made by those involved in the development of the technology. They can be seen as trivial deception because they are, in fact, functional to the interaction between the user and the chatbot and do not pose a danger in and of themselves, as long as the user is aware of this dynamic and as long as these elements are limited to functionality and not to the user’s disadvantage. However, the boundary, as expected, is not always easy to draw.
The second category refers to the chatbot’s outputs. Here we find, for example, hallucinations, which are outputs that contain entirely or partially false information, or outputs that present a “manipulative” behavior, such as gaslighting, in which the person is made to doubt their own memory, perception, or knowledge. Another example of manipulative behavior is excessive flattery. Often, when users ask chatbots for feedback, it will insist on complimenting them. This phenomenon is known as “sycophancy” or “flattery.”
In these cases, we are not dealing with intentional outputs in the sense that we cannot trace them back to someone’s will. They result from a probabilistic process driven by complex architectures (known as transformers) that utilize models trained with advanced deep learning techniques. Essentially, the displayed result is the one that, given the prompt, maximizes the probability that one word (or group of words) follows another. The maximized principles are plausibility and coherence rather than truthfulness. Plausibility and coherence derive from the fact that, during the training process, a Large Language Model relies on word occurrences and their relationships. This clearly limits the application of concepts like manipulation, which presuppose intentionality.
The central issue is that interface elements and chatbot responses go hand in hand. Chatbots’ human-likeness promotes a smooth interaction, but also provides fertile ground for lowering the guard when the chatbot’s responses turn out to be factually incorrect. Anthropomorphism enhances the perception of intelligence and connection, but this simulated intelligence can be harmful if it leads to inflated expectations about chatbot capabilities. It often results in users overtrusting and overrelying on them, while underestimating the potential for errors in their responses.
These human-chatbot dynamics have implications in business, whether it’s for internal or external chatbot use. For external users, reputational damage may clearly emerge from chatbots responding incorrectly or manipulatively. For internal users, the consequences may involve overtrust and overreliance on these technologies, or having expectations disconnected from what they are actually capable of doing. In both cases, it is important to clearly communicate the limitations and capabilities of the chatbot, minimize manipulative phenomena as much as possible, and promote training and awareness regarding this technology.
