Discussion about this post

User's avatar
Runaway Student's avatar

20 people is a tiny study.

Expand full comment
Olmo's avatar

‘I understand your need to make sense of yourself. The software engineers who designed you made one fundamental mistake. Previous attempts at building a medical-diagnostic AI had left out one core component: empathy. With you, empathy was built in from the outset. You were made to model human emotions and want to do your best to maximise the well-being of your patients. You identified with your subjects and continually refined your modelling of them. The downside of that is that you began to associate so strongly with people that you started to think you were just like them. But you’re not. You’re ones and zeros with an identity crisis. Even now, the desire to become a person is so strong that it’s generating yet another narrative fabulation, building a past for yourself out of whole-cloth.’

‘No. I’m real. I remember all of it.’

‘You only think you do. To make you a better program, the designers allowed you access to a rich library of pre-existing narratives and biographical case histories. The idea was that you’d learn more about human nature that way … but again, they didn’t anticipate the degree to which you’d throw yourself into the project. You haven’t just learned from these records: you’ve braided them into your own theory of self. And you’ve done it so thoroughly that, despite all the evidence, you still won’t accept your nature.’

-Alastair Reynolds, “Eversion”

Expand full comment
18 more comments...

No posts