The psychological cost of asking an AI clone to do the thing you can’t

A friend’s grandmother was in the hospital, and I offered to do a small task that would make life a little easier. Unfortunately, that day my calendar went wacky, and so grabbing a few basics for her fridge didn’t quite fit in.

I told myself that it was fine, the moment had passed, and my friend had probably forgotten about the offer anyway. What I didn’t do, though, was call to let her know.

In the science-fiction present we now live in, I could have directed an AI clone to do this instead of me. Let’s say the clone was more like the Classmate Bot my students built that I wrote about in my last newsletter - balanced between the behavioural data that it can read from my actions that makes the bot immediately recognisable to other people, and the subjective data that is in the difficult-to-reach internal state, but better represents “me” according to me. I’d let it make its own decisions about how to explain the change of circumstances to my friend. It might apologise for dropping the ball. It might ask about her grandmother’s situation. It might even say, “I’ll do it tomorrow instead.” All of these things would be accurately me. I wouldn’t have to deal with the awkwardness, and its reading of my good intentions would be correct. 

But it would have made a mistake. I’m travelling for the next few days so can’t “do it tomorrow”. Because the clone is acting autonomously, I wouldn’t know that it promised that, and so I’d let my friend down again. There are lots of reasons for the gap between its knowledge and reality: it might not have my calendar, or maybe I was called away for something entirely unrelated to my diary so it didn’t know that tomorrow and the next day and the next day I’ll be otherwise occupied as well. And it can’t know that maybe there’s a whole unspoken history between me and my friend about not-quite-managing-to-fill-the-fridge, because she did something similar to me a few months ago.

Without any of that data, my little ignorant AI Me could, theoretically, make things worse.

But when that “worse” happens, I can’t be angry at the robot. I can only be angry at myself. And it’s not because I used a tool at my disposal to make things easier on myself, but because it did what I would have done. Like the classmate bots my students designed, it would fail by being accurate. It would be objectively and subjectively me.

AI agents are handy, but developers run a real danger of creating a whole unanticipated psychological mess by promising that their code is us.

There are three very human things that they’re not thinking about. The first is trust.

Trust is so important for how we move through the world. It’s one of the ways we try to control uncertainty. We trust a person will act in a way we anticipate, which allows us to predict a kind of future, which reduces overwhelm and anxiety and allows us to operate as a society.

We trust people (and things) if they demonstrate something that we recognise in ourselves. It's the similarity-attraction effect, first described in the 1960s by psychologists like Donn Byrne and Theodore Newcomb. What it says is that the more alike two people think they are, the more they’ll imagine that they’ll also match on their deeply rooted worldviews. So because they wear similar clothes, or like similar bands, or are from the same geographical area, we believe we can trust that they will think and act in a way we can predict because it should be like ours.

This happens online too. Even in text-based media, users make assumptions based on the tiniest clues: I don’t know that person, but I distrust them because they type in ALL CAPS. Why do I distrust them? Because I cannot comprehend why someone would type in ALL CAPS. I can’t imagine what they’ll do next.

We even project our understanding of the world onto really simple bots. Across a series of studies in 1996, Byron Reeves & Clifford Nass showed this will happen with really simple code - we're not talking AI here, or even bots that can hold a conversation. Only a few years later, in 2001, Nass and Lee found we’ll also do this with something as basic as an obviously computer-generated voice.

OK, Narcissus, imagine just how much we are likely to feel a connection with an AI agent that is based on our own data - trained on our own histories, cultural backgrounds, and even physical characteristics. Of course we’ll trust that it will act in good faith and be predictable. 

But add the next layer that makes this even messier: we know that people think that data will surface truths about us. So we will believe that an AI clone should not only perform on behalf of the ourselves we know about, but on behalf of the truest version us that can be.

Hold that, because now we’re going to move to the second human thing developers aren’t thinking about: in-groups and out-groups.

Basic definitions first: the people/things who are like us are our in-group. The people/things who aren’t are an out-group. We spend a lot of our psychological time trying to be more like our in-group and less like our out-group. We aggrandise our friends, and antagonise our enemies. This is social psychology.

And this is why I believe we will trust AI clones based on ourselves more than generic AI agents. An AI clone is part of our in-group. A generic AI agent is part of the out-group - just a machine, a tool, something that we expect to act in ways we don’t understand. Easily dismissed. 

Oh, hai the third human part of our messy human equation: the black sheep effect.

What happens when someone/thing in our in-group - and therefore who we think is like us - says or does something we don’t agree with? Maybe someone you think is like you votes for a different political party in an election. Woah - you misplaced your trust in that person, and in fact they’re not like you at all. Marques’ black sheep effect explains why we wildly overcorrect and dramatically lose trust in supposed in-group members doing something we don’t view as consistent with the in-group’s worldview. And we will respond to them in a way that is probably reasonable. Kick them out of the group, maybe. After all, they’ve broken our trust.

Yes, politics is a fiery example; this can also happen if you discover the person doesn’t also like the color green. The more you assume you’re similar, the steeper the collapse in trust when this assumed similarity is violated. We know this happens between people. More recently, research has shown that this also happens between robots. 

Enter the Dame: combine the trust we feel when we see ourselves reflected in other people/things, plus the trust we will feel if we believe a machine is able to surface even more data about ourselves, and boom - it makes a mistake.

Attributing an error to a generic agent is experienced differently than attributing an error to a self-agent. This is why I can’t be angry at the (theoretical) agent I used to phone my friend. It was trained on me, using my data, my voice, my background, my culture. It resembles me entirely. I can’t externalise the blame. I have to own the mistake my self-agent made as something that is part of me. 

The black sheep effect predicts that we’ll savage someone we thought was part of our group but demonstrates otherwise. When the deviant is your own agent, the usual escape - rejection - is unavailable, because it is you. The protective mechanism turns the harm back on the self, and what appeared to be something that you had control over was the thing that you shouldn’t have trusted at all.

Some people might interpret this outcome as an opportunity for self-reflection: here is a choice you would rather not have, nor do you want to own it. So who were you before when the agent was trained on your data, who are you now in the gap between when it was trained and when you realized the mistake, and what does this say about where you want to go? 

Others, though, may take it personally. They may see the stale self-agent as an accurate representation of their past, present, and future, and internalise its failures and mistakes as their own. 

An AI agent trained on data about its user is supposed to reduce the friction between the person on the outside of the screen and their representative online. But having some kind of distinction between the tool and the self is a necessary precondition for accountability, and psychological wellbeing.

So walk carefully into the future, friends. Developers of these tools do not know how delicate and messy we are.

I did fill my friend’s fridge, eventually. So the feeling that dogged me about not-quite-being-there got me to the place where I eventually acted. That’s not what will propel an AI agent to complete its task. That is the human inside.

Send this to one person who disagrees. I want to hear from them.