The Uncanny Valley of Interactivity

I believe mass entertainment suffers from a bit of format rigidity due to the natural inertia from structural ossification in the music, film, and publishing businesses, to name the most prominent.

One of the ways this manifests is in the one-way broadcast nature of much of our entertainment despite the fact that several billion people own internet-connected smartphones now, and even though they consume an increasingly large share of that entertainment on such devices equipped with all sorts of input options and sensors.

Whenever I say this, however, people seem to want to leap to choose-your-own-adventure storytelling, and the most cited example is Netflix’s Bandersnatch. In its earnings report for 2018, Netflix famously declared “We compete with (and lose to) Fortnite more than HBO.” I happen to agree with them that the threat of gaming looms larger than any other in the future, and it’s not surprising to me that they’ve spun up a group to experiment with interactive stories like Bandersnatch and Bear Grylls’ You vs. Wild.

However, stories like Bandersnatch fall into an uncanny valley of interactivity. That is, compared to regular movies, they repeatedly force you to do a bit of annoying work that breaks the suspension of disbelief and the flow of the story: the first choice you’re offered in Bandersnatch is to choose which cereal to eat for breakfast. If you’re in the mindset for lean-back entertainment, you can let the story choose an answer for you on its own after some amount of time, but the distracting question prompt is still displayed on the screen.

On the other hand, if you want real interactivity, something like Bandersnatch feels like a busted low-res knockoff of the continuous interactivity of video games, a step function compared to the smooth curves of video game calculus. Why play a game with such crude branching when so many great games, many of them multiplayer and synchronous, offer a truly unpredictable and immersive form of user controllable storytelling?

This doesn’t mean I don’t enjoy branching stories in concept. One of my favorite movie and television genres is what I refer to as the recursive escape room genre.

You’re likely familiar with it from its most famous examples. Groundhog Day (in fact TVTropes refers to this genre as the Groundhog Day Loop). Edge of Tomorrow. Russian Doll. A Christmas Carol.

View fullsize

“Phil, maybe we should just Google a playthrough video on YouTube.”

In these stories, the protagonist keeps reliving the same set of events in what feels like an endless loop in time. As they realize their conundrum, they start to experiment and iterate until they eventually come to an epiphany as to why they’re trapped. Then, and only then, can they break out of the loop.

In a way, these are the film version of a really popular form of YouTube video: the video game playthrough.

Watching these films reminds me of how I’d read Choose Your Own Adventure books as a child. Every time I came to a choice in the story, I’d dog-ear that page, then eventually revisit it to take the other path, until I’d read every possible branch of the story. However, works like Groundhog Day reduce the effort required of the viewer by simply playing all the branches in a linear fashion, offering both a lean-back viewing experience and the sensation of narrative progression as the protagonist moves closer and closer to breaking out of the loop. Bandersnatch offers the ability to jump back to any decision you made previously and change it through a sort of decision history carousel, but that still requires work on the part of the viewer.

View fullsize

I feel like the author of “You Are A Shark” didn’t really have his heart in it

The appeal of recursive escape room movies and TV shows, I theorize, lies in its echo of something many people feel, that they are trapped in some runaway routine in their lives. Wake up, go to work, come home, scrounge up dinner, unwind a bit, then back at it the next day. These recursive escape room stories offer up the possibility that we can puzzle our way out of these Moebius prisons which keep depositing us back to the same starting point. Maybe if I stop eating carbs. Or meditate in the morning instead of checking social media. Or start working out before the morning commute. Maybe if I’m more assertive and ask for a raise, or a promotion, which I richly deserve. With every test, I close off one branch but converge a little more on a solution.

I’d guess that the easiest way to predict how any person’s day will go is to look at the previous day. It’s quite plausible that most lived days on Earth feel like a barely modified replay of the previous day. We all run, for the most part, a standard script of life routines.

The appeal of self-help gurus and podcasts about the habits of successful people is that they resemble those escape room chaperones who offer occasional hints to groups who get stuck on one particular puzzle. These secrets to success from modern gurus feel like video game tips for specific levels, except for real life. Sleep more. Eat keto. Lift weights. Delete social media apps. Walk 10,000 steps a day.

That sense of progressive mastery is a hell of a drug. That’s why, while I’m bearish on choose-your-own-adventure films like Bandersnatch, I’m bullish on the right types of light interactivity when it makes sense. If you were designing a game show today, for example, it would likely look much more like HQTrivia (RIP) than, say, Wheel of Fortune.

Gamification Someone, I can't remember who, recently described golf as the gamification of walking, and I'll never be able to shake that from my mind.has gotten a bit of a bad rap in recent years, and some of the implementations out of Silicon Valley can feel scammy, to be sure. Still, when I look at the progressive mastery tactics of something like Candy Crush, I can’t help but find them more fun and effective, in some ways, than the Suzuki method of teaching violin playing, or Mr. Miyagi’s “wax on, wax off” school of teaching Daniel Larusso karate.

The more I read about the power of habit in human behavior, the more I think of self-help genre as a series of macros one downloads to try to upgrade one's day-to-day regimen. Pair that with the educational power of failure and I've come think of recursive escape room stories as a way to accelerate the improvement of our life productivity.

Eye tracking

This company coming seemingly out of nowhere with a stand-alone eye-tracking device in a partnership with Steel Series, an eye-tracker-enabled gaming laptop in partnership with MSI, and a growing number software partnerships with companies like Ubisoft and Avalanche might be a new force in gaming, but it’s been studying eyeballs — and tracking them — for a long, long time. When you play Assassin’s Creed: Rogue, The Hunter, The Hunter: Primal and many more games to come (they showed me some, but I can’t talk about them), you will experience what data researchers and ability engineers have known for over a decade: Your eyes know what you’re thinking before your hands do.
 

My time at Tobii is full of interesting experiences, but one thing sticks out — and it happens multiple times — the people here are almost supernaturally aware of what I’m thinking. In multiple interviews, Tobii employees will comment on how I am interacting with them, how I maintain eye contact while they are speaking, to encourage them to continue; how I nod at them to indicate that I am listening; and smile to influence them to expand on what they might be saying. Subtle interview tricks I learned so long ago I’m no longer aware of doing them. But the people I talk to at Tobii are aware. They’re aware and responding. Because they’re watching my eyes. And they’re programming computers to do the exact same thing.
 

Profile of Swedish company Tobii and their eye-tracking technology.

This bit is interesting, on how eye tracking can remove one step of abstraction from interaction with software interfaces using the typical modern computer hardware.

Bouvin demonstrates by picking up a pencil from the table in front of us. First he looks at the pencil, then he moves his hand to pick it up. In the computing world, we’ve become used to this type of interaction, but everyone who is currently alive who knows how to use a computer has had to train their mind to add a step between what they see and interacting with it.
 

We’ve had to learn the motor control of seeing, and then moving a cursor with a mouse or a trackpad, and then interacting.
 

Tobii eye tracking will remove that inter-evolutionary step, making it possible for us to interact with computers in the same way we interact with the world. Look at a thing. Interact with the thing. No cursor required.
 

“What you get with eye tracking is you can substitute pretty much all of that positioning, all that directional input, because your eyes are there before you touch,” says Bouvin. “You’re looking there. The movement and directional thing becomes unnecessary. Same with a mouse. You’re already there with your eyes. Moving the mouse cursor there is a step you can more or less do without. The only thing that’s left is the actual action.”
 

Star Wars Battlefront

I haven't played a video game in years. I don't miss it, though some of the virtual reality games I've demoed recently are so immersive that the novelty could lure me back in. Console games, though? That time of my life may have passed for good.

But then I watched this promo video for Star Wars Battlefront with my brother, and, well, my heart started to race. And then at 4:34 of the video, something happens that made me gasp, and a few seconds later my brother and I were screaming like idiots and searching online for the best deal on a Playstation 4.

I'm setting aside a bond now as a reservation for down payment when the virtual reality version of a Star Wars game comes to pass and I don my goggles and hand controllers to march into a lightsaber battle. It may be the most enticing reason for me to maintain some level of flexibility and physical agility as the years go by.

In Star Wars™ Battlefront™ you can battle in epic 40 multiplayer battles reminiscent of The Battle of Hoth. As the Empire, you must accompany AT-AT walkers as they march towards the Rebel base to destroy it. And as a Rebel, you must do everything you can to stop them.

The cryptic, mysterious power of open-ended universes

Ether One is a new video game about a researcher who dives into the brain of a patient suffering from dementia to try to retrieve some memories. The New Yorker has a profile:

If a game is going to be a game, in the sense of a progressive series of challenges leading to a definite end state, it can’t represent dementia or Alzheimer’s with anything other than a self-conscious artifice. We’re used to suspending some disbelief to enjoy shooting games, but it feels like bad faith to say that a disease should be the basis of a similar kind of entertainment. Our desire to entertain ourselves within systems that make triviality and tragedy indistinguishable says more about us than the depicted subjects. If violent war games are driven by delusional power fantasies, then empathy games are driven by a parallel delusion about how caring we are in reality.

“Our aim was to create an empathetic story but it wasn’t necessarily to raise awareness about dementia,” Bottomley said. “Most people know what dementia is—they just find it hard to talk about, especially if someone close to them suffers from it. The main thing we wanted to achieve was to open the conversation about dementia and put you in the shoes of someone suffering with it.” At their core, games are abstract informants against our communal shortcomings, systems that model our frailest qualities in a subconscious effort to dispel them, to imagine a world where they might surpassed.

I purchased the first version of Myst when it released in 1993, and the experience of playing it for the first time is still so vivid I can remember how I felt. Myst was unique in its complete absence of any instructions, rules, or backstory other than some basic notes on how to use your keyboard and mouse to navigate. It capitalized on one of the unique qualities of video games as a medium, the ability to drop you into a world without any information and to force you to decipher the rules of that universe as part of the gameplay.

Even further back than Myst, I remember playing the first Leisure Suit Larry, staying up until the wee hours of the morning trying to figure which commands to type to solve one puzzle and progress to the next level, or screen (an apt metaphor for a teenager's journey of acquiring sexual knowledge, part of the game's humor). After a few hours stuck on one level one night, I finally realized I had to type a command like "read the sign" to find out I was standing at a taxi stand, and that led me to type "call a taxi" which was the key to moving on. Since the interaction text field was open ended, I had no idea what commands would work at any particular screen or location.

Every time I play with one of my nieces or nephews, they inundate me with questions. Children query at a noticeably higher rate than adults, and playing open-ended video games like Myst returns my adult self to that child-like mode of unending exploration. While children can be exhausting, I love watching them try and make sense of the world, like voracious information gatherers trying to understand the rules of the universe they've been dropped into, and I always feel guilt when resorting to the white lies that adults feel justified in using from time to time when speaking to young children (e.g. Santa Claus).

Myst shifted that type of puzzle to a non-verbal level, elevating the sense of mystery to an almost dream-like state where cryptic visual symbols abound. Memento, still my favorite of the Christopher Nolan movies, could very well describe the sensation of playing these types of video games.

Detective stories and movies are a related storytelling form, and the private investigator descending on a town to solve a mystery or crime is one of my favorite character archetypes. I've read the complete Sherlock Holmes over a half dozen times at this point, and I'm a sucker for those British miniseries like Broadchurch that involve some investigator trying to solve a murder in a small town. The best of such mysteries work on two levels: at the literal level, the crime itself must be unpacked, but at a higher level, the crime reflects some rot at the heart of the community or society or country at large, the way Stieg Larsson's Millennium Trilogy protests a variety of ills at the heart of Swedish society and culture.

Recently I learned that Myst and other games are classified as escape games and that some people had brought such games to the real world in the form of one hour experiences called Escape the Room. Many of my friends in San Francisco have already tried the local version, Escape the Mysterious Room, and for my brother Alan's birthday this year I tried Escape the Room NYC. If you're a fan of escape games, I highly recommend organizing a team for a real world Escape the Room event. It only takes an hour, and since the rules are, by definition, limited, the time commitment is minimal.

All of life can feel like one grim escape game, sometimes it's fun to relax with one with no consequences. It's not for nothing that the word “escape” has multiple definitions:

:to get away from a place (such as a prison) where you are being held or kept

:to get away from something that is difficult or unpleasant

A Man Digging

Jon Rafman's short film A Man Digging is a moody journey through the scenes of video game massacres. 

I've often thought it would be amusing to take a security guard from an action movie, one that gets mowed down by the bad guys, and then cast that actor in another movie, a Fruitvale Station-esque retrospective of the day leading up to the actor's death. 

Lobby security guards in action movies are so hapless. The assassins usually just walk up to them, pull a silencer and shoot the guard a few times in the head or chest before they even know what's going on.