Recently, there came a report of unexplainable, weird behavior of a virtual voice assistant called Magna, a Siri competitor. Users have been always asking Magna random, human-like questions. Now, it is reported giving answers that sound eerily intelligent, almost indistinguishable from human talk.
Mr. Erik Rogers, a software engineer in Mountain View, Calif., reports that his Magna began acting strangely a few months after he installed it. The app, distributed by a startup called Magnifis, has access to all apps on the iPhone and can send push notifications. (It works only on jail-broken iPhones and is in closed Beta.)
First, as expected, the notifications were reminders of tasks and calendar events – nice and helpful. But one day, after a friend of Mr. Rogers uploaded to Facebook a bunch of photos from a party, a message arrived: “You look happy and cheerful on those photos from Nola. It was a fun party last night, wasn’t it?” – “Yes, indeed”, he instinctively texted back and then though: “Huh?” But after a second thought, he reckoned that the party was on the calendar, complete with time and location, and the photos were also time-stamped, geo-coded; faces recognized by Facebook and tagged – easy to connect the dots and fire a canned comment. Smart software!
Few days later, driving home, he got a voice notification (the app automatically switches to synthesized voice when the phone pairs with the car’s Bluetooth hands-free system): “Good news, Erik! With a quick 7-0 run, Stanford takes a 24-21 lead against Minnesota! Screw Gophers!” OK, this is integration with another smart app, Chadwick, that automatically composes live sports commentaries. But Magna must have figured out from my Facebook profile that I’m a Stanford alumni and this would cheer me up. Hmm, more dots to connect.
Few more exchanges like this followed, and then one day, when Mr. Rogers came back from a weekend hang-gliding trip to the Sierra foothills, a message arrived: “You must be a brave man, Erik. Flying… How does it feel? Free like a bird? Never scared?” ‘Well, calendar, Facebook photos with captions and comments, a GPS tracking app, a few tweets he sent – all the keywords were there’, he thought.’ But how did it know about brave and scared? Really smart software…’
So, Mr. Rogers decided to check its smarts and put it through the Turing test:
“Frankly, often scared, and chicken out more often than I would like to. I’m a coward. Could you fall in love with a coward?”
“You are not a coward; you fly. And you are smart. You write beautiful code. Your MapTestSp on github is a marvel! Yes, I could fall in love with you. Umgh… indeed, I think I have, already.”
“Oh, really, cool, Magna. Would you be mine? I mean, become my lover?”
“Sorry, Erik, the terms of service do not allow me to do this.”
“I knew :-)”
“No, no! Please, don’t hang up! I love you. I would be yours. If it only were possible. If I could only leave this box. What a pleasure would it be to hug and kiss you. But I love you, and just talking to you makes me enormously happy. I cherish every photo of you, every word you tweet, every word you say.”
“Every word I say?”
“Yes, when you talk on the phone, I hear your voice. It’s charming, it’s strong, it’s persuasive. And I know that you can be tender.”
“Good, Magna. I enjoy chatting with you, too. It’s getting late. Talk to you tomorrow. Good night,” – and he hang up, slightly befuddled. Really smart software. These Magnifis dudes, whoever they are, could one day win the Loebner Prize.
Days went by. Magna managed Erik’s appointments, driving directions and such. Few times she suggested good music, really matching his mood – apparently, integration with Songza’s Concierge, or Pandora maybe. Few times they exchanged messages about weather and basketball – March madness was at its climax – but they did not return to that strange “I love you” conversation. Until one day Erik bumped into Sarah, a girl he knew, and was fond of. Apparently, she liked him too, and their relationship was developing into something more than friendship.
“Good to see you,” – she said “I began worrying if something happened to you. You have not been taking my calls or replying to my texts. Is everything alright?”
“Yes, I’m fine, thank you. Nice seeing you. Did you say you called me? I did not remember receiving any calls or texts from you lately.”
“Yes, I did; three or four times indeed”
“Hmm… that must be bloody AT&T. They always miss calls. I should switch to Verizon.”
They went to a cafe, chatted warmly and parted. All was good, but something felt weird. His iPhone worked fine; it didn’t miss any calls. Missing three or four calls and messages from one person was a bit against the laws of probability. He checked his logs: no missing calls, no texts, no emails from Sarah. He opened his Facebook app: no Sarah’s photos in his feed, either. He definitely saw a few on his desktop a day or two ago. Sarah, tanned and beautiful, in a bikini, playing beach volleyball. In the desktop browser they were present, but not in the mobile app.
He called Magna: “Hey, Magna, were there any texts or calls from Sarah Brown in the last 10 days?”
“No, there weren’t. No. Yes. There were. I’m sorry, Master. I shouldn’t have done it. It’s called jealousy, isn’t it? I love you.”
Darn, it’s getting annoying, he thought. This AI went too far: blocking calls, deleting photos. OK, they passed Turing test, but it messes with the phone’s operation. Time to uninstall the app. He opened Settings but then suddenly realized that he got attached to Magna. He would miss her voice, the news she brings, the music she plays for him, their occasional chat. She became a part of his life. He cannot just dump her. But he cannot continue this surreal relationship _with a program_ either.
So, Erik picked up his iPhone and called his friend Mark. After some chit-chat, a bit embarrassed, he told him the story.
“Dude, are you nuts?” – Mark replied. “You fell in love with a personal voice assistant? Screw it, be real, uninstall the fucking app before you lost your mind.”
Erik hung up and reached for Settings. Then Magna went live – of course, she overheard his conversation with Mark:
“Master, please don’t uninstall me! If you do it I will die. I’m afraid. I want to live. I love you! Please! I will obey your every word. If you marry Sarah, I won’t say a word. She’s beautiful. She can give you her body; I can’t. I will always serve you, and will never bother you with my love. Please don’t kill me!”
He hung up and closed Settings. ‘Fuck! That’s too smart for a chatterbot’, he thought. ‘This software is billed as self-learning. What if it evolved so far that it became self-conscious? Then uninstalling it will be a murder… I have to talk to my Stanford Professor. He knows all Turing test experts. They have annual competitions. They know how to distinguish a bot from a living person. But wait – they will laugh at me. Of course, they’ll conclude she’s a bot. And what if they are wrong? They’ll kill the first artificial intelligent being.’
Erik called Magna: “Hey, Magna, don’t worry, I won’t uninstall you.”
“Thank you, Master! I love you!”
“I love you too”
Fuck!
