My students' AI clones got everything right about them. And that's what went wrong.

I teach a course on a Master’s degree for people who want to create the next generation of digital technology. It’s at NYU in their Interactive Telecommunications Programme - ITP. I love teaching there - the students are artists and programmers and designers, marketers, former Disney Imagineers, and even English lit grads. I learn so much and get so inspired by the weirdness that they create, and how I get to provoke them into being even weirder.

In one of my favourite assignments, I ask my students to design an AI version of one of their classmates. They have to deliver a bot, an interface, or an app that should be identifiable as the target classmate in a blind test by the rest of the room. The high level concept I’m trying to teach is the distinction between subjectivity and objectivity - what it feels like to be a person versus what is necessary to be identified as that person - and which actually matters in the final test.

By watching them work, I’ve learned a whole lot about whether who we feel we are actually matters anymore in today’s technologically enhanced world.

There’s a lot going around in the ether about AI lately (you may have noticed), and it really feels like my day job has bled into my every day. 

It has to be a decade since I made the first of several radio programs where I interviewed Eugenia Kuyda, founder of the AI-companion service Replika; this was when the company was still called Luka, when she was still riding the roller coaster of bereavement and trying to hold on to the living pulse of her recently-deceased best friend Roman, a bot that was trained on their text messages. That’s how it all started. A pure, human act of two people being completely themselves to each other, thrust into something that has hard edges. Eugenia’s bot was a creative coping solution for the soft-and-jaggy parts of us that are impossible to articulate. Ask poets how hard that is. That’s their job too.

Since then, I’ve spoken with Eugenia several times - always for programs about the unexpected consequences of trying to fit a human into a box made of binary code, how it pushes up against our felt boundaries of who we think we are and the social contracts we have with the rest of the world. I was talking about this recently with a friend who’s in charge of  “AI transformation” in a multinational corporation. ‘We never have time to think of the important stuff,’ he told me. The stuff that comes downstream of the decisions he’ll be making today. That’s what I get to do, and it’s unbelievably rewarding. It’s what I get to teach my students too.

One of the things he’s been talking about with his bosses is in the almost-possible conversations I’m hearing everywhere people are bullish about AI: AI twins, or AI agents you can train on your own data to be you. And this is where the unintended consequences of the subjective and objective really explode.

The first part of my students’ Classmate Bot project is to write a character sketch that describes what they already know about their assigned person. I ask them to draw from their own experience, and what they can find about the person online. What do they look like, what do they do, what do they want -> all based on behavioural data, of course, the stuff that can be observed. Some descriptions are more flowery, some are clean bullet points. The aim is to imagine someone else being able to identify them from this description alone.

The second part of the task is sharing this sketch with their person. This is never an easy meeting. Everyone approaches it with varying degrees of narcissistic curiosity, and then ends up feeling like an empty shell - unseen.

Being “seen”. Such a contemporary concept, and hundreds - if not thousands - of lines have been written about how we try and fail to do this in the digital world.

Feeling respected and understood has been a fundamental human need since forever, and the impact of not is losing something my field of psychology argues is a driver of everything we do: our self-esteem. The clearest examples of this comes from studies of stigmatised identities - people who, for whatever reason (disability, social status, sexuality, health status, ethnicity, etc) are devalued by society. In Glynis Breakwell’s 1986 book, Coping with Threatened Identities, she introduced a framework for analysing how people cope with being stereotyped - grouped into a collective of (in this case, marginalised) traits. Losing the identity they feel they are (their positives) causes people to try to hide the thing they’re threatened by (the negatives), or over-compensate by criticising other people who are also part of their group. Not being seen can lead to the erosion of self-worth, persistent anxiety, depression, isolation, and a loss of a connection to who we think we are. 

The digital world is absolutely a platform of self-expression that can lead to connecting with our people, but it’s also lately become a series of shopfront identities that expose a chasm that separates us from the self we feel from the self our behaviour projects. There’s psychological element to this, but it’s also because of what tools we have to hand, and how well we can bend them to our will. 

The third part of the Classmate Bot assignment is collaboration. This is the part where my students try to pin down the subjective experience - the bit that wants to be ‘seen’. So, together, the sketch-writer and the sketch-target correct what was written down on the paper. First pass, use what’s there - you got this right, you got this wrong; here are the corrections. 

But then to go deeper, the designer interviews the target to find out what it feels like to be the person they’re going to build into the bot. Their questions are designed to access their classmate’s lived experience - why did you do X, what did it feel like when Y happened, how do you go about doing Z, tell me about a time when ABC. The balance is to get as much information as they can about their target’s inside life with the kind of information that they can translate into an interactive design. 

The final step of the process is taking all of this data, and turning that into their Classmate Bot.

The conditions for Eugenia Kuyda’s Roman bot were different than what my students have. It was built out of private conversations between two people who were very close, who knew each other’s histories, and who felt they were able to be their actual - rather than their ideal - selves. So the quality of the data Kuyda had to work with was way better at accessing Roman’s subjective self than what my students have. 

But the bot was never Roman. It was a shadow of him. It wasn’t built to be him. It was built to provide comfort. A very human objective, but one that’s entirely subjective on the part of the person who used it. There was no Roman to test whether it matched his inner self. That didn’t matter. The bot was for Eugenia and for her alone.

Inevitably, the Classmate Bots err towards the objective person they’re meant to represent. That makes sense: the aim of the assignment is to have other people recognise who has been built into the machine, and it’s easier to design what someone does rather than, for example, how they felt when their grandmother was sick in hospital, and how that experience has affected them since. But what that becomes is a stereotype. A sketch.  

And when my students try out the designs that are meant to represent them, they know they’re not wrong, exactly. The knowledge and the descriptions are accurate. But there’s no depth. The characters feel cold, unrepresentative, and as one student fed back, a “billboard” version of themselves. They don’t feel seen.

This isn’t something that better tools will solve. In fact, it’s amplified by the companies trying to create the digital twins out of only behavioural data. They’re all doing the same assignment as my students are, and they’re all grading themselves on the same measure: recognizability. Of course they are - observable can be benchmarked. It can be scored. But they are bypassing the subjective internal experience that also defines who a person is, basing their definition of ‘human’ on only behavioral stuff. As I asked at the outset - does that even matter? 

The goal for AI twins companies is to build something that will represent a user enough that it can act on its own and wear that user’s name in the wild, for vastly different audiences, to do vastly different tasks. But the Classmate Bot experiment shows that recognisability is the wrong criterion.

My students don’t want their Classmate Bots out in the world acting on their behalf. They don’t trust the bot will accurately represent them in all the ways that they have to show up. And their bots are built out of conversations that produce more intimate knowledge than the stuff about them that’s out there online. 

Who we feel we are does still matter. Flattening a self isn’t just inaccurate, it can do damage. I’m watching an industry sprinting forward by optimising what Glynis Breakwell documented forty years ago as harmful, and calling it a benchmark.

Being seen can’t be scored, though, so not measuring it isn’t negligence. The shape of the incentive is the only thing that can move. Unless, the AI twins industry is suggesting that we do.

Go on - forward this to one person who'd argue with me.