Drawing invisible boundaries in conversational interfaces

One of the things anyone who has worked on textual conversation interfaces, like chatbots, will tell you is that the challenge is dealing with the long tail of crazy things people will type. People love to abuse chatbots. Something about text-based conversation UI's invites Turing tests. Every game player remembers the moment they first abandoned their assigned mission in Grand Theft Auto to start driving around the city crashing into cars, running over pedestrians, just to exercise their freedom and explore just what happens when they escape the plot mechanic tree.

However, this type of user roaming or trolling happens much less with voice interfaces. Sure, the first time a user tries Siri or Alexa or whatever Google's voice assistant is called (it really needs a name, IMO, to avoid inheriting everything the word "Google" stands for), they may ask something ridiculous or snarky. However, that type of rogue input tends to trail off quickly, whereas it doesn't in textual conversation UI's.

I suspect some form of the uncanny valley and blame the affordances of text interfaces. Most text conversation UI's are visually indistinguishable from those of a messaging UI used to communicate primarily with other human beings. Thus it invites the user to probe its intelligence boundaries. Unfortunately, the seamless polish of the UI isn't matched by the capabilities of chatbots today, most of which are just dumb trees.

On the other hand, none of the voice assistants to date sounds close to replicating the natural way a human speaks. These voice assistants may have more human timbre, but the stiff elocution, the mispronunciations, the frequent mistakes in comprehension, all quickly inform the user that what they are dealing with is something of quite limited intelligence. The affordances draw palpable, if invisible, boundaries in the user's mind, and they quickly realize the low ROI on trying anything other than what is likely to be in the hard-coded response tree. In fact, I'd argue that the small jokes that these UI's insert, like answering random questions like "what is the meaning of life?" may actually set these assistants up to disappoint people even more by encouraging more such questions the assistant isn't ready to answer (I found it amusing when Alexa answered my question, "Is Jon Snow dead?" two seasons ago, but then was disappointed when it still had the same abandoned answer a season later, after the question had already been answered by the program months ago).

The same invisible boundaries work immediately when speaking to one of those automated voice customer service menus. You immediately know to speak to these as if you're addressing an idiot who is also hard of hearing, and the goal is to complete the interaction as quickly as possible, or to divert to a human customer service rep at the earliest possible moment.

[I read on Twitter that one shortcut to get to a human when speaking to an automated voice response system is to curse, that the use of profanity is often a built-in trigger to turn you over to an operator. This is both an amusing and clever design but also feels like some odd admission of guilt on the part of the system designer.]

It is not easy, given the simplicity of textual UIs, to lower the user's expectations. However, given where the technology is for now, it may be necessary to erect such guardrails. Perhaps the font for the assistant should be some fixed-width typeface, to distinguish it from a human. Maybe some mechanical sound effects could convey the robotic nature of the machine writing the words, and perhaps the syntax should be less human in some ways, to lower expectations.

One of the huge problems with voice assistants, after all, is that the failures, when they occur, feel catastrophic from the user perspective. I may try a search on Google that doesn't return the results I want, but at least something comes back, and I'm usually sympathetic to the idea that what I want may not exist in an easily queryable form on the internet. However, though voice assistant errors occur much less frequently than before, when they do, it feels as if you're speaking to a careless design, and I mean careless in all sense of the word, from poorly crafted (why didn't the developer account for this obvious query) and uncaring (as in emotionally cold).  

Couples go to counseling over feeling as if they aren't being heard by each other. Some technology can get away with promising more than they can deliver, but when it comes to tech that is built around conversation, with all the expectations that very human mode of communication has accrued over the years, it's a dangerous game. In a map of the human brain, the neighborhoods of "you don't understand" and "you don't care" share a few exit ramps.

10 more browser tabs

Still trying to clear out browser tabs, though it's going about as well as my brief flirtation with inbox zero. At some point, I just decided inbox zero was a waste of time, solving a problem that didn't exist, but browser tab proliferation is a problem I'm much more complicit in.

1. Why the coming-of-age narrative is a conformist lie

From a more sociological perspective, the American self-creation myth is, inherently, a capitalist one. The French philosopher Michel Foucault theorised that meditating and journalling could help to bring a person inside herself by allowing her, at least temporarily, to escape the world and her relationship to it. But the sociologist Paul du Gay, writing on this subject in 1996, argued that few people treat the self as Foucault proposed. Most people, he said, craft outward-looking ‘enterprising selves’ by which they set out to acquire cultural capital in order to move upwards in the world, gain access to certain social circles, certain jobs, and so on. We decorate ourselves and cultivate interests that reflect our social aspirations. In this way, the self becomes the ultimate capitalist machine, a Pierre Bourdieu-esque nightmare that willingly exploits itself.
 
‘Growing up’ as it is defined today – that is, as entering society, once and for all – might work against what is morally justifiable. If you are a part of a flawed, immoral and unjust society (as one could argue we all are) then to truly mature is to see this as a problem and to act on it – not to reaffirm it by becoming a part of it. Classically, most coming-of-age tales follow white, male protagonists because their integration into society is expected and largely unproblematic. Social integration for racial, sexual and gender minorities is a more difficult process, not least because minorities define themselves against the norm: they don’t ‘find themselves’ and integrate into the social context in which they live. A traditional coming-of-age story featuring a queer, black girl will fail on its own terms; for how would her discovering her identity allow her to enter a society that insists on marginalising identities like hers? This might seem obvious, but it very starkly underscores the folly of insisting on seeing social integration as the young person’s top priority. Life is a wave of events. As such, you don’t come of age; you just age. Adulthood, if one must define it, is only a function of time, in which case, to come of age is merely to live long enough to do so.
 

I've written about this before, but almost always, the worst type of film festival movie is about a young white male protagonist coming of age. Often he's quiet, introverted, but he has a sensitive soul. As my first year film school professor said, these protagonists are inert, but they just "feel things." Think Wes Bentley in American Beauty filming a plastic bag dancing in the wind for fifteen minutes with a camcorder, then showing it to a girl as if it's Citizen Kane.

If they have any scars or wounds, they are compensated for with extreme gifts. Think Ansel Elgort in Baby Driver; cursed with tinnitus since childhood, he listens to music on a retro iPod (let's squeeze some nostalgic product placement in here, what the hell, we're also going to give him a deaf black foster father to stack the moral cards in his favor, might as well go all the way) and is, that's right, the best getaway driver in the business.

Despite having about as much personality as a damp elephant turd, their beautiful souls are both recognized and extracted by a trope which this genre of film invented just for this purpose, the manic pixie dream girl.

[Nathan Rabin, who invented the term manic pixie dream girl, has since disavowed the term as sometimes misogynist, and it can be applied too broadly like a hammer seeking nails, but it doesn't undo the reality that largely white male writing blocs, from guilds to writer's rooms, aren't great at writing women or people of color with deep inner lives.]

This is tangential to the broader point, that the coming-of-age story as a genre is, in and of itself, a lie. It reminds me of the distinction between Finite and Infinite Games, the classic book from James Carse. The Hollywood film has always promised a finite game, and thus it's a story that must have an ending. Coming-of-age is an infinite game, or at least until death, and so we should all be skeptical of its close-ended narrative.

(h/t Michael Dempsey)

2. Finite and Infinite Games and The Confederate

This isn't a browser tab, really, but while I'm on the topic of Carse's Finite and Infinite Games, a book which provides a framework with which so much of the world can be bifurcated, and while I'm thinking about the white male dominated Hollywood profession, I can't help but think of the TV project The Confederate, by the showrunners of Game of Thrones.

"White people” is seen by many whites as a pejorative because it lowers them to a racial class whereas before they were simply the default. They are not accustomed to having spent their entire lives being named in almost every piece of culture as a race, the way women, people of color, and the union of the two are, every single day, by society and culture.

All Lives Matter retort to Black Lives Matter is to pretend that we're all playing the same finite game when almost everyone who are losers in that game know it is not true. Blacks do not feel like they “won” the Civil War; every day today they live with the consequences and the shadow of America's founding racism, every day they continue to play a game that is rigged against them. That is why Ta Nehisi Coates writes that the question of The Confederate is a lie, and that only the victors of this finite game of America would want to relitigate the Civil War in some Alt History television show for HBO. It's as if a New England Patriot fan asked an Atlanta Falcons fan to watch last year's Super Bowl again, with Armie Hammer playing Tom Brady.

“Give us your poor, your huddled” is a promise that the United States is an infinite game, an experiment that struggles constantly towards bettering itself, evening the playing field, such that even someone starting poor and huddled might one day make a better life and escape their beginning state. That is why Stephen Miller and other white nationalists spitting on that inscription on the Statue of Liberty is so offensive, so dangerous.

On society, Carse writes:

The prizes won by its citizens can be protected only if the society as a whole remains powerful in relation to other societies. Those who desire the permanence of their prizes will work to sustain the permanence of the whole. Patriotism in one or several of its many forms (chauvinism, racism, sexism, nationalism, regionalism) is an ingredient in all societal play. 
 
Because power is inherently patriotic, is is characteristic of finite players to seek a growth of power in a society as a way of increasing the power of a society.
 

Colin Kaepernick refusing to stand for the National Anthem is seen as unpatriotic by many in America, including the wealthy white owners of such teams, which is not surprising, as racism is a form of patriotism, per Carse, and part and parcel of American society when defined as a finite game.

Donald Trump and his large adult sons are proof of just how powerful the inheritance of title and money are in America, and the irony that they are elected by those who feel that successive rounds of finite games have started to be rigged against them is not lost on anyone, not even, I suspect, them. One could argue they need to take a lesson from those oppressed for far longer as to how a turn to nihilism works out in such situations.

Those attacking Affirmative Action want to close off the American experiment and turn it into a series of supposedly level finite games because they have accumulated a healthy lead in this game and wish to preserve it in every form.

White nationalists like Trump all treat America as not just a finite game, but a zero sum finite game. The idea of immigrants being additive to America, to its potential, its output, is to treat America as an infinite game, open-ended. The truth lies, as usual, between the poles, but closer to the latter.

Beware the prophet who comes with stories of zero-sum games, or as Jim Collins once wrote, beware the "tyranny of the or." One of my definitions of leadership is the ability to turn zero-sum into positive sum games.

3. Curb Your Enthusiasm is Running Out of People to Offend

Speaking of fatigue with white male protagonists:

But if Larry David’s casual cruelty mirrors the times more than ever, the show might still fit awkwardly in the current moment. Watching the première of Season 9 on Sunday night, I kept thinking of a popular line from George Costanza, David’s avatar on “Seinfeld”: “You know, we’re living in a society!” Larry, in this first episode of the season, seems to have abandoned society altogether. In the opening shot, the camera sails over a tony swath of L.A., with no people and only a few cars visible amid the manicured lawns and terra-cotta roofs. It descends on Larry’s palatial, ivy-walled house, where he showers alone, singing Mary Poppins’s “A Spoonful of Sugar” and bludgeoning a bottle of soap. (Its dispenser pump is broken—grounds for execution under the David regime.) He’s the master of his domain, yes, but only by default: no one else is around.
 
“Curb” has always felt insulated, and a lot of its best jokes are borne of the fact that Larry’s immense wealth has warped his world view over the years. (On the most recent season he had no compunction about spending a princely sum on Girl Scout Cookies, only to rescind the order out of spite.) But the beginning of Season 9 offers new degrees of isolation. Like a tech bro ensconced in a hoodie and headphones, Larry seems to have removed himself almost entirely from public life. Both “Curb” and “Seinfeld” like to press the limits of etiquette and social mores, but the latter often tested these on subway cars and buses, in parks or on the street. Much of “Curb,” by contrast, unfolds in a faceless Los Angeles of air-conditioned mansions, organic restaurants, and schmoozy fund-raisers, a long chain of private spaces. The only time Larry encounters a true stranger, it’s in the liminal zone between his car and the lobby of Jeff’s office. She’s a barber on her way to see Jeff at work—even haircuts happen behind closed doors now.
 

Groundhog Day, one of the great movies, perhaps my favorite Christmas movie of all time, has long been regarded a great Buddhist parable

Groundhog Day is a movie about a bad-enough man—selfish, vain, and insecure—who becomes wise and good through timeless recurrence.
 

If that is so, then Curb Your Enthusiasm is its dark doppelganger, a parable about the dark secret at the heart of American society, that no person, no matter how selfish, vain, and petty, can suffer the downfall necessary to achieve enlightenment, if he is white and a man. 

In this case, he is a successful white man in Hollywood, Larry David, and each episode of Curb Your Enthusiasm is his own personal Groundhog Day. Whereas Bill Murray wakes up each morning to Sonny and Cher, trapped in Punxsutawney, Pennsylvania, around small town people he dislikes, in a job he feels superior to, Larry David wakes up each morning in his Los Angeles mansion, with rewards seemingly only proportionate to the depths of his pettiness and ill humor. Every episode, he treats all the friends and family around him with little disguised disdain, and yet the next episode, he wakes up in the mansion again.

Whereas Bill Murray eventually realizes the way to break out of his loop is to use it for self-improvement, Larry David seems to be striving to fall from grace by acting increasingly terrible and yet finds himself back in the gentle embrace of his high thread count sheets every morning.

Curb Your Enthusiasm has its moments of brilliance in its minute dissection of the sometimes illogical and perhaps fragile bonds of societal goodwill, and its episode structure is often exceedingly clever, but I can't help watching it now as nothing more than an acerbic piece of performance art, with all the self absorption that implies.

Larry David recently complained about the concept of first world problems, which is humorous, as it's difficult to think of any single person who has done as precise a job educating the world on what they are.

[What about Harvey Weinstein and Louis C.K., you might ask? Aren't they Hollywood royalty toppled from lofty, seemingly untouchable perches? The story of how those happened will be the subject of another post, because the mechanics are so illuminating.]

4. Nathan for You

I am through season 2 of Nathan for You, a Comedy Central show that just wrapped its fourth and final season. We have devalued the term LOL with overuse, but no show has made me literally laugh out loud by myself, on the sofa, as this, though I've grinned in pleasure at certain precise bits of stylistic parody of American Vandal.

Nathan Fielder plays a comedic version of himself. In the opening credits, he proclaims:

My name is Nathan Fielder, and I graduated from one of Canada's top business schools with really good grades [NOTE: as he says this, we see a pan over his transcript, showing largely B's and C's]. Now I'm using my knowledge to help struggling small business owners make it in this competitive world.
 

If you cringed while watching a show like Borat or Ali G, if you wince a bit when one of the correspondents on The Daily Show went to interview some stooge, you might believe Nathan For You isn't, well, for you. However, the show continues to surprise me.

For one thing, it's a deeply useful reminder of how difficult it is for physical retailers, especially mom and pop entrepreneurs, to generate foot traffic. That they go along with Fielder's schemes is almost tragic, but more instructive.

For another, while almost every entrepreneur is the straight person to Fielder's clown, I find myself heartened by how rarely one of them just turns him away outright. You can see the struggle on each of their faces, as he presents his idea and then stares at them for an uncomfortably long silence, waiting for them to respond. He never breaks character. Should they just laugh at him, or throw him out in disgust? It almost never happens, though one private investigator does chastise Fielder for being a complete loser.

On Curb Your Enthusiasm, Larry David's friends openly call him out for his misanthropy, yet they never abandon him. On Nathan For You, small business owners almost never adopt Fielder's ideas at the end of the trial. However, they almost never call him out as ridiculous. Instead, they try the idea with a healthy dose of good nature at least once, or at least enough to capture an episode's worth of material.

In this age of people screaming at each other over social media, I found this reminder of the inherent decency of people in face to face situations comforting and almost reassuring. Sure, some people are unpleasant both online and in person, and some people are pleasant in person and white supremacists in private.

But some people try to see the best in each other, give others the benefit of the doubt, and on such bonds a civil society are maintained. That this piece of high concept art could not fence in the humanity and real emotion of all the people participating, not even that of Fielder, is a bit of pleasure in this age of eye-rolling cynicism.

[Of course, these small business owners are aware a camera is on them, so the Heisenberg Principle of reality television applies. That a show like this, which depend on the subjects not knowing about the show, lasted four full seasons is a good reminder of how little-watched most cultural products are in this age of infinite content.]

BONUS CONTENT NO ONE ASKED FOR: Here is my Nathan for You idea: you know how headline stand-up comedians don't come on stage to perform until several lesser known and usually much lousier comics are trotted out to warm up the crowd? How, if you attend the live studio taping of a late night talk show like The Daily Show or The Tonight Show, some cheesy comic comes out beforehand to get your laugh muscles loose, your vocal chords primed? And when the headliner finally arrives, it comes as sweet relief?

What if there were an online dating service that provided such a warm-up buffoon for you? That is, when you go on a date, before meeting your date, first the service sends in a stand-in who is dull, awkward, a turn off in every way possible? But a few minutes into what seems to be a disastrous date, you suddenly show up and rescue the proceedings?

It sounds ridiculous, but this is just the sort of idea that Nathan for You would seem to go for. I haven't watched seasons 3 and 4 yet, so if he does end up trying this idea in one of those later episodes, please don't spoil it for me. I won't even be mad that my idea was not an original one, I'll be so happy to see actual footage of it in the field.

5. The aspect ratio of 2:00 to 1 is everywhere

I first read the case for 2:00 to 1 as an aspect ratio when legendary cinematographer Vittorio Storaro advocated for it several years ago. He anticipated a world where most movies would have a longer life viewed on screens at home than in movie theaters, and 2:00 to 1, or Univisium, is halfway between the typical 16:9 HDTV aspect ratio and Panavision, or 2:35 to 1.

So many movies and shows use 2:00 to 1 now, and I really prefer it to 16:9 for most work.

6. Tuning AIs through captchas

Most everyone has probably encountered the new popular captcha which displays a grid of photos and asks you to identify which contain a photo of a store front. I just experienced it recently signing up for HQTrivia. This breed of captcha succeeds the wave of captchas that showed photos of short strings of text or numbers and asked you to type in what you saw, helping to train AIs trying to learn to read them. There are variants of the store front captcha: some ask you to identify vehicles, others to identify street signs, but the speculation is that Google uses these to train the "vision" of its self-driving cars.

AI feels like magic when it works, but underrated is the slow slog to take many AI's from stupid to competent. It's no different than training a human. In the meantime, I'm looking forward to being presented with the captcha that shows two photos, one of a really obese man, the other of five school children, with this question above them: "If you had to run over and kill the people in one of these photos, which would you choose?"

7. It's Mikaela Shiffrin profile season, with this one in Outside and this in the New Yorker

I read Elizabeth Weil's profile of Shiffrin in Outside first:

But the naps: Mikaela not only loves them, she’s fiercely committed to them. Recovery is the most important part of training! And sleep is the most important part of recovery! And to be a champion, you need a steadfast loyalty to even the tiniest and most mundane points. Mikaela will nap on the side of the hill. She will nap at the start of the race. She will wake up in the morning, she tells me after the gym, at her house, while eating some pre-nap pasta, “and the first thought I’ll have is: I cannot wait for my nap today. I don’t care what else happens. I can’t wait to get back in bed.”
 
Mikaela also will not stay up late, and sometimes she won’t do things in the after­noon, and occasionally this leads to more people flipping out. Most of the time, she trains apart from the rest of the U.S. Ski Team and lives at home with her parents in Vail (during the nine weeks a year she’s not traveling). In the summers, she spends a few weeks in Park City, Utah, training with her teammates at the U.S. Ski and Snowboard Center of Excellence. The dynamic there is, uh, complicated. “Some sports,” Mikaela says, “you see some athletes just walking around the gym, not really doing anything, eating food. They’re first to the lunchroom, never lifting weights.”
 

By chance, I happened to be reading The Little Book of Talent: 52 Tips for Improving Your Skills by Daniel Coyle, and had just read tips that sounded very familiar to what was mentioned here.

More echoes of Coyle's book in The New Yorker profile:

My presumption was that her excellence was innate. One sometimes thinks of prodigies as embodiments of peculiar genius, uncorrupted by convention, impossible to replicate or reëngineer. But this is not the case with Shiffrin. She’s as stark an example of nurture over nature, of work over talent, as anyone in the world of sports. Her parents committed early on to an incremental process, and clung stubbornly to it. And so Shiffrin became something besides a World Cup hot shot and a quadrennial idol. She became a case study. Most parents, unwittingly or not, present their way of raising kids as the best way, even when the results are mixed, as such results usually are. The Shiffrins are not shy about projecting their example onto the world, but it’s hard to argue with their findings. “The kids with raw athletic talent rarely make it,” Jeff Shiffrin, Mikaela’s father, told me. “What was it Churchill said? Kites fly higher against a headwind.”
 

So it wasn't a real surprise to finally read this:

The Shiffrins were disciples of the ten-thousand-hours concept; the 2009 Daniel Coyle book “The Talent Code” was scripture. They studied the training methods of the Austrians, Alpine skiing’s priesthood. The Shiffrins wanted to wring as much training as possible out of every minute of the day and every vertical foot of the course. They favored deliberate practice over competition. They considered race days an onerous waste: all the travel, the waiting around, and the emotional stress for two quick runs. They insisted that Shiffrin practice honing her turns even when just skiing from the bottom of the racecourse to the chairlift. Most racers bomb straight down, their nonchalance a badge of honor.
 

Coyle's book, which I love for its succinct style (it could almost be a tweetstorm if Twitter had slightly longer character limits, each tip is averages one or two paragraphs long), is the books I recommend to all parents who want their kids to be really great at something, and not just sports.

Much of the book is about the importance of practice, and what types of practice are particularly efficient and effective.

Jeff Shiffrin said, “One of the things I learned from the Austrians is: every turn you make, do it right. Don’t get lazy, don’t goof off. Don’t waste any time. If you do, you’ll be retired from racing by the time you get to ten thousand hours.”
 
“Here’s the thing,” Mikaela told me one day. “You can’t get ten thousand hours of skiing. You spend so much time on the chairlift. My coach did a calculation of how many hours I’ve been on snow. We’d been overestimating. I think we came up with something like eleven total hours of skiing on snow a year. It’s like seven minutes a day. Still, at the age of twenty-two, I’ve probably had more time on snow than most. I always practice, even on the cat tracks or in those interstitial periods. My dad says, ‘Even when you’re just stopping, be sure to do it right, maintaining a good position, with counter-rotational force.’ These are the kinds of things my dad says, and I’m, like, ‘Shut up.’ But if you say it’s seven minutes a day, then consider that thirty seconds that all the others spend just straight-lining from the bottom of the racecourse to the bottom of the lift: I use that part to work on my turns. I’m getting extra minutes. If I don’t, my mom or my coaches will stop me and say something.”
 

Bill Simmons recently hosted Steve Kerr for a mailbag podcast, and part I is fun to hear Kerr tell stories about Michael Jordan. Like so many greats, Jordan understood that the contest is won in the sweat leading up to the contest, and his legendary competitiveness elevated every practice and scrimmage into gladiatorial combat. As Kerr noted, Jordan single-handedly was a cure for complacency for the Bulls. 

He famously broke down some teammates with such intensity in practice that they were driven from the league entirely (remember Rodney McCray?). Everyone knows he once punched Steve Kerr and left him with a shiner during a heated practice. The Dream Team scrimmage during the lead in to the 1992 Olympics, in which the coaches made Michael Jordan one captain, Magic Johnson the other, is perhaps the single sporting event I most wish had taken place in the age of smartphones and social media.

What struck me about the Shiffrin profiles, something Coyle notes about the greats, is how many of the lives of the great ones are unusually solitary, spent in deliberate practice on their own, apart from teammates. It's obviously amplified for individual sports like tennis and skiing and golf, but even for team sports, the great ones have their own routines. Not only is it lonely at the top, it's often lonely on the way there.

8. The secret tricks hidden inside restaurant menus

Perhaps because I live in the Bay Area, it feels as if the current obsession is with the dark design patterns and effects of social apps. But in the scheme of things, many other fields whose work we interact with daily have many more years of experience designing to human nature. In many ways, people designing social media have a very naive and incomplete view of human nature, but the power of the distribution of ubiquitous smartphone and network effects have elevated them to the forefront of the conversation.

Take a place like Las Vegas. Its entire existence is testament to the fact that the house always wins, yet it could not exist if it could not convince the next sucker to sit down at the table and see the next hand. The decades of research into how best to part a sucker from his wallet makes the volume of research among social media companies look like a joke, even if the latter isn't trivial.

I have a sense that social media companies are similar to where restaurants are with menu design. Every time I sit down at a new restaurant, I love examining the menus and puzzling over all the choices with fellow diners, as if having to sit with me over a meal isn't punishment enough. When the waiter comes and I ask for an overview of the menu, and recommendations, I'm wondering what dishes the entire experience is meant to nudge me to order.

I'm awaiting the advent of digital and eventually holographic or A/R menus to see what experiments we'll see. When will we have menus that are personalized? Based on what you've enjoyed here and other restaurants, we think you'll love this dish. When will we see menus that use algorithmic sorting—these are the most ordered dishes all-time, this week, today? People who ordered this also ordered this? When will see editorial endorsements? "Pete Wells said of this dish in his NYTimes review..."

Not all movies are worth deep study because not all movies are directed with intent. The same applies to menus, but today, enough menus are put through a deliberate design process that it's usually a worthwhile exercise to put them under the magnifying glass. I would love to read some blog that just analyzes various restaurant menus, so if someone starts one, please let me know.

9. Threat of bots and cheating looms as HQ Trivia reaches new popularity heights

When I first checked out HQ Trivia, an iOS live video streaming trivia competition for cash prizes, the number of concurrent viewers playing, displayed on the upper left of the screen, numbered in the hundreds. Now the most popular of games, which occur twice a day, attract over 250K players. In this age where we've seen empires built on exploiting the efficiencies to be gained from shifting so much of social intimacy to asynchronous channels, it's fun to be reminded of the unique fun of synchronous entertainment.

What intrigues me is not how HQ Trivia will make money. The free-to-play game industry is one of the most savvy when it comes to extracting revenue, and even something like podcasts points the way to monetizing popular media with sponsorships, product placement, etc.

What's far more interesting is where the shoulder on the S-curve is. Trivia is a game of skill, and with that comes two longstanding issues. I've answered, at most, 9 questions in a row, and it takes 12 consecutive right answers to win a share of the cash pot. I'm like most people in probably never being able to win any cash.

This is an issue faced by Daily Fantasy Sports, where the word "fantasy" is the most important word. Very soon after they became popular, DFS were overrun by sharks submitting hundreds or thousands of lineups with the aid of computer programs, and some of those sharks worked for the companies themselves. The "fantasy" being sold is that the average person has a chance of winning.

As noted above in my comment about Las Vegas, it's not impossible to sell people on that dream. The most beautiful of cons is one the mark willingly participates in. People participate in negative expected value activities all the time, like the lottery, and carnival games, and often they're aware they'll lose. Some people just participate for the fun of it, and a free-to-play trivia game costs a player nothing other than some time, even if the expected value is close to zero.

A few people have asked me whether that live player count is real, and I'm actually more intrigued by the idea it isn't. Fake it til you make it is one of the most popular refrains of not just Silicon Valley but entrepreneurs everywhere. What if HQ Trivia just posted a phony live player count of 1 million tomorrow? Would their growth accelerate even more than it has recently? What about 10 million? When does the marginal return to every additional player in that count go negative because people feel that there is so much competition it's not worth it? Or is the promise of possibly winning money besides the point? What if the pot scaled commensurate to the number of players; would it become like the lottery? Massive pots but long odds?

The other problem, linked to the element of skill, is cheating. As noted in the article linked above, and in this piece about the spike in Google searches for answers during each of the twice-a-day games, cheating is always a concern in games, especially as the monetary rewards increase. I played the first game when HQ Trivia had a $7,500 cash pot, and the winners each pocketed something like $575 and change. Not a bad payout for something like 10 minutes of fun.

Online poker, daily fantasy sports, all are in constant battle with bots and computer-generated entries. Even sports books at casinos have to wage battle with sharks who try to get around betting caps by sending in all sorts of confederates to put down wagers on their behalf.

I suspect both of these issues will be dampeners on the game's prospects, but more so the issue of skill. I already find myself passing on games when I'm not with others who also play or who I can rope into playing with me. That may be the game's real value, inspiring communal bonding twice a day among people in the same room.

People like to quip that pornography is the tip of the spear when it comes to driving adoption of new technologies, but I'm partial to trivia. It is so elemental and pure a game, with such comically self-explanatory rules, that it is one of the elemental forms or genres of gaming, just like HQ Trivia host Scott Rogowsky is some paragon of a game-show host, mixing just the right balance of cheesiness and snarkiness and effusiveness needed to convince all the players that any additional irony would be unseemly.

10. Raising a teenage daughter

Speaking of Elizabeth Weil, who wrote the Shiffrin profile for Outside, here's another of her pieces, a profile of her daughter Hannah. The twist is that the piece includes annotations by Hannah after the fact.

It is a delight. The form is perfect for revealing the dimensions of their relationship, and that of mothers and teenage daughters everywhere. In the interplay of their words, we sense truer contours of their love, shaped, as they are, by two sets of hands.

[Note, Esquire has long published annotated profiles, you can Google for them, but they are now all locked behind a paywall]

This format makes me question how many more profiles would benefit from allowing the subject of a piece to annotate after the fact. It reveals so much about the limitations of understanding between two people, the unwitting and witting lies at the heart of journalism, and what Janet Malcolm meant, when she wrote, in the classic opening paragraph of her book The Journalist and the Murderer, "Every journalist who is not too stupid or too full of himself to notice what is going on knows that what he does is morally indefensible."

Helpful tip for data series labels in Excel

I've never received as many emails from readers as I did for my most recent post on line graphs, analytics, Amazon, Excel, and Tufte, among other things. It turns out that countless consultants, bankers, and analysts still wake up in a cold sweat at the recollection of spending hours formatting graphs in Excel. I opened every last Excel spreadsheet you sent me, apparently we all have one or another lying around as a souvenir of our shared trauma.

[It's fun to hear directly from my readers, I encourage more of it. When so much of online discourse is random strangers performing drive-by violence, or chucking Molotov cocktails at you, it's somewhat old-fashioned and comforting to receive a friendly email. It reminds me a lot of the early days of the Internet, a more idyllic time, when Utopian dreams of the power of this technology hadn't been crushed by the darkness in the souls of mankind.]

Many readers just shared their stories of Excel frustration, but a few offered helpful tips. One class of these was just a recommendation to try alternative programs for charting, like Tableau, ggplot, or D3. I have not had time to play with any of these, though I remember working with Tableau briefly at a company that had purchased a license. I'm not familiar enough with any of these to offer any meaningful comparison to Excel. 

I still hope that someone who works on Excel will just upgrade their default charting options because of the sheer number of Microsoft Office users. Most companies don't license Tableau, and many may not have the inclination or time to learn to use ggplot and D3, though the latter, in particular, seems capable of generating some beautiful visualizations. In contrast, I can't recall any company I worked at that didn't offer me a Microsoft Office license if I wanted one.

A few readers mentioned they wrote macros to automate some or most of the formatting tricks I set out in my post. Maybe at some point this will become an open source tool, which I'd love. If I spot one I'll share it here. Excel users are a tight knit community.

A couple readers offered a similar solution to my problem with generating dynamic data series labels, but my favorite, the most comprehensive option, came from a reader named Jeffrey, a finance associate.

It's a clever little hack. Here's how it works:

  1. Fill in your data table in Excel as normal, with data series in rows, time periods (or whatever the x-axis will be) in columns.
  2. Add one duplicate row for each data series below those data series in your table, and name each of those the same as the data series above, in order. For example, if your data series are U.S., Japan, and Australia, you now add three more rows, named U.S., Japan, and Australia.
  3. For these dummy data series, leave all the data cells blank except the last one, which in my example was the year 2014. For those three cells, just use a formula so that that cell equals the corresponding last data point from the corresponding data series above. So, for the dummy data series for Australia, my formula for 2014 just points to the value in the cell for 2014 for the actual Australia data series.
  4. Now create the chart for your data table, including both the actual and dummy data series. In the resulting line graph, you now have six data series instead of three, but the three dummy data series just have a single data point, the last one, which overlaps the corresponding last point in the actual data series.
  5. Do all the visual hacks I mentioned in my previous post. When it comes to your three dummy data series, instead of displaying the actual data label, select "Series Name" in the Format Data Labels menu.

That's it! Now you have data labels that will move with any change in the last data point of your three data series.

It's a hack, so it's not perfect. You still have to update your table each time you add to it, moving the dummy pointers one cell to the right. And selecting the data series when it exactly overlaps the actual data series can be tricky. But the thing is, all my old ways of setting up the charts were hacks, too, and this one saves the work of moving data series labels whenever the scale of the graph is changed.

It would be easier if Excel just offered a way to display the data labels and the series name for every data series. Maybe someday? The communication arc of the universe is long, and I like to think it bends towards greater efficiency.

Thanks Jeffrey! Also, thanks to my old coworker Dave who, more than any of my other readers, sends me precise copy-edits to my posts. I wish I were a better editor of my own work, but familiarity breeds myopia.

Remove the legend to become one

When I started my first job at Amazon.com, as the first analyst in the strategic planning department, I inherited the work of producing the Analytics Package. I capitalize the term because it was both a serious tool for making our business legible, and because the job of its production each month ruled my life for over a year.

Back in 1997, analytics wasn't even a real word. I know because I tried to look up the term, hoping to clarify just I was meant to be doing, and I couldn't find it, not in the dictionary, not on the internet. You can age yourself by the volume of search results the average search engine returned when you first began using the internet in force. I remember when pockets of wisdom were hidden in eclectic newsgroups, when Yahoo organized a directory of the web by hand, and later when many Google searches returned very little, if not nothing. Back then, if Russians wanted to hack an election, they might have planted some stories somewhere in rec.arts.comics and radicalized a few nerds, but that's about it.

Though I couldn't find a definition of the word, it wasn't difficult to guess what it was. Some noun form of analysis. More than that, the Analytics Package itself was self-describing. Literally. It came with a single page cover letter, always with a short preamble describing its purpose, and then jumped into a textual summary of the information within, almost like a research paper abstract, or a Letter to Shareholders. I like to think Jeff Bezos' famous company policy, instituted many years later, banning Powerpoint in favor of written essays, had some origins in the Analytics Package cover letter way back when. The animating idea was the same: if you can't explain something in writing to another human, do you really understand it yourself?

My interview loop at Amazon ended with an hour with the head of recruiting at the time, Ryan Sawyer. After having gone through a gauntlet of interviews that included almost all the senior executives, and including people like Jeff Bezos and Joy Covey, some of the most brilliant people I've ever met in my life, I thought perhaps the requisite HR interview would be a letup. But then Ryan asked me to explain the most complex thing I understood in a way he'd understand. It would be good preparation for my job.

What was within the Analytics Package, that required a written explanation? Graphs. Page after page of graphs, on every aspect of Amazon's business. Revenue. Editorial. Marketing. Operations. Customer Service. Headcount. G&A. Customer sentiment. Market penetration. Lifetime value of a customer. Inventory turns. Usually four graphs to a page, laid out landscape.

The word Package might seem redundant if Analytics is itself a noun. But if you saw one of these, you knew why it was called a Package. When I started at Amazon in 1997, the Analytics Package was maybe thirty to forty pages of graphs. When I moved over to product management, over a year later, it was pushing a hundred pages, and I was working on a supplemental report on customer order trends in addition. Analytics might refer to a deliverable or the practice of analysis, but the Analytics Package was like the phone book, or the Restoration Hardware catalog, in its heft.

This was back in the days before entire companies focused on building internal dashboards and analytical tools, so the Analytics Package was done with what we might today consider as comparable to twigs and dirt in sophistication. I entered the data by hand into Excel tables, generated and laid out the charts in Excel, and printed the paper copies.

One of the worst parts of the whole endeavor was getting the page numbers in the entire package correct. Behind the Analytics Package was a whole folder of linked spreadsheets. Since different charts came from different workbooks, I had to print out an entire Analytics Package, get the ordering correct, then insert page numbers by hand in some obscure print settings menu. Needless to say, ensuring page breaks landed where you wanted them was like defusing a bomb.

Nowadays, companies hang flat screen TVs hanging on the walls, all them running 24/7 to display a variety of charts. Most everyone ignores them. The spirit is right, to be transparent all the time, but the understanding of human nature is not. We ignore things that are shown to us all the time. However, if once a month, a huge packet of charts dropped on your desk, with a cover letter summarizing the results, and if the CEO and your peers received the same package the same day, and that piece of work included charts on how your part of the business was running, you damn well paid attention, like any person turning to the index of a book on their company to see if they were mentioned. Ritual matters.

The package went to senior managers around the company. At first that was defined by your official level in the hierarchy, though, as most such things go, it became a source of monthly contention as to who to add to the distribution. One might suspect this went to my head, owning the distribution list, but in fact I only cared because I had to print and collate the physical copies every month.

I rarely use copy machines these days, but that year of my life I used them more than I will all the days that came before and all the days still to come, and so I can say with some confidence that they are among the least reliable machines ever made by mankind.

It was a game, one whose only goal was to minimize pain. A hundred copies of a hundred page document. The machine will break down at some point. A sheet will jam somewhere. The ink cartridge will go dry. How many collated copies do you risk printing at once? Too few and you have to go through the setup process again. Too many and you risk a mid-job error, which then might cascade into a series of ever more complex tasks, like trying to collate just the pages still remaining and then merging them with the pages that were already completed. [If you wondered why I had to insert page numbers by hand, it wasn't just for ease of referencing particular graphs in discussion; it was also so I could figure out which pages were missing from which copies when the copy machine crapped out.]

You could try just resuming the task after clearing the paper jam, but in practice it never really worked. I learned that copy machine jams on jobs of this magnitude were, for all practical purposes, failures from which the machine could not recover.

I became a shaman to all the copy machines in our headquarters at the Columbia building. I knew which ones were capable of this heavy duty task, how reliable each one was. Each machine's reliability fluctuated through some elusive alchemy of time and usage and date of the last service visit. Since I generally worked late into every night, I'd save the mass copy tasks for the end of my day, when I had the run of all the building's copy machines.

Sometimes I could sense a paper jam coming just by the sound of machine's internal rollers and gears. An unhealthy machine would wheeze, like a smoker, and sometimes I'd put my hands on a machine as it performed its service for me, like a healer laying hands on a sick patient. I would call myself a copy machine whisperer, but when I addressed them it was always a slew of expletives, never whispered. Late in my tenure as analyst, I got budget to hire a temp to help with the actual printing of the monthly Analytics Package, and we keep in touch to this date, bonded by having endured that Sisyphean labor.

My other source of grief was another tool of deep fragility: linked spreadsheets in Excel 97. I am, to this day, an advocate for Excel, the best tool in the Microsoft Office suite, and still, if you're doing serious work, the top spreadsheet on the planet. However, I'll never forget the nightmare of linked workbooks in Excel 97, an idea which sounded so promising in theory and worked so inconsistently in practice.

Why not just use one giant workbook? Various departments had to submit data for different graphs, and back then it was a complete mess to have multiple people work in the same Excel spreadsheet simultaneously. Figuring out whose changes stuck, that whole process of diffs, was untenable. So I created Excel workbooks for all the different departments. Some of the data I'd collect myself and enter by hand, while some departments had younger employees with the time and wherewithal to enter and maintain the data for their organization.

Even with that process, much could go wrong. While I tried to create guardrails to preserve formulas linking all the workbooks, everything from locked cells to bold and colorful formatting to indicate editable cells, no spreadsheet survives engagement with a casual user. Someone might insert a column here or a row there, or delete a formula by mistake. One month, a user might rename a sheet, or decide to add a summary column by quarter where none had existed before. Suddenly a slew of #ERROR's show up in cells all over the place, or if you're unlucky, the figures remain, but they're wrong and you don't realize it.

Thus some part of every month was going through each spreadsheet and fixing all the links and pointers, reconnecting charts that were searching for a table that was no longer there, or more insidiously, that were pointing to the wrong area of the right table.

Even after all that was done, though, sometimes the cells would not calculate correctly. This should have been deterministic. That's the whole idea of a spreadsheet, that the only error should be user error. A cell in my master workbook would point at a cell in another workbook. They should match in value. Yet, when I opened both workbooks up, one would display 1,345 while the other would display 1,298. The button to force a recalculation of every cell was F9. I'd press it repeatedly. Sometimes that would do it. Sometimes it wouldn't. Sometimes I'd try Ctrl - Alt - Shift - F9. Sometimes I'd pray.

One of the only times I cried at work was late one night, a short time after my mom had passed away from cancer, my left leg in a cast from an ACL/MCL rupture, when I could not understand why my workbooks weren't checking out, and I lost the will, for a moment, to wrestle it and the universe into submission. This wasn't a circular reference, which I knew could be fixed once I pursued it to the ends of the earth, or at least the bounds of the workbook. No, this inherent fragility in linked workbooks in Excel 97 was a random flaw in a godless program, and I felt I was likely the person in the entire universe most fated to suffer its arbitrary punishment.

I wanted to leave the office, but I was too tired to go far on my crutches. No one was around the that section of the office at at that hour. I turned off the computer, turned out the lights, put my head down on my desk for a while until the moment passed. Then I booted the PC back up, opened the two workbooks, and looked at the two cells in question. They still differed. I pressed F9. They matched. 

Most months, after I had finished collating all the copies of the Analytics Package, clipping each with a small, then later medium, and finally a large binder clip, I'd deliver most copies by hand, dropping them on each recipient's empty desk late at night. It was a welcome break to get up from my desk and stroll through the offices, maybe stop to chat with whoever was burning the midnight oil. I felt like a paper boy on his route, and often we'd be up at the same hour.

For all the painful memories that cling to the Analytics Package, I consider it one of the formative experiences of my career. In producing it, I felt the entire organism of our business laid bare before me, its complexity and inner working made legible. The same way I imagine programmers visualizing data moving through tables in three dimensional space, I could trace the entire ripple out from a customer's desire to purchase a book, how a dollar of cash flowed through the entire anatomy of our business. I knew the salary of every employee, and could sense the cost of their time from each order as the book worked its way from a distributor to our warehouse, from a shelf to a conveyor belt, into a box, then into a delivery truck. I could predict, like a blackjack player counting cards in the shoe, what % of customers from every hundred orders would reach out to us with an issue, and what % of those would be about what types of issues.

I knew, if we gained a customer one month, how many of their friends and family would become new customers the next month, through word of mouth. I knew if a hundred customers made their first order in January of 1998, what % of them would order again in February, and March, and so on, and what the average basket size of each order would be. As we grew, and as we gained some leverage, I could see the impact on our cash flow from negotiating longer payable days with publishers and distributors, and I'd see our gross margins inch upwards every time we negotiated better discounts off of list prices.

What comfort to live in the realm of frequent transactions and normal distributions, a realm where the laws of large numbers was the rule of law. Observing the consistency and predictability of human purchases of books (and later CDs and DVDs) each month was like spotting some crystal structure in Nature under a microscope. I don't envy companies like Snapchat or Twitter or Pinterest, social networks who have gone public or likely have to someday, companies who play in the social network business, trying to manage investor expectations when their businesses are so large and yet still so volatile, their revenue streams even more so. It is fun to grow with the exponential trajectory of a social network, but not fun if you're Twitter trying to explain every quarter why you missed numbers again, and less fun when you have to pretend to know what will happen to revenue one quarter out, let alone two or three.

At Amazon, I could see our revenue next quarter to within a few percentage points of accuracy, and beyond. The only decision was how much to tell Wall Street we anticipated our revenue being. Back then, we always underpromised on revenue; we knew we'd overdeliver, the only question was how much we should do so and still maintain a credible sense of surprise on the next earnings call.

The depth of our knowledge of our own business continues to exceed that of any company I've worked at since. Much of the credit goes to Jeff for demanding that level of detail. No one can set a standard for accountability like the person at the top. Much credit goes to Joy and my manager Keith for making the Analytics Package one of the strategic planning department's central tasks. That Keith pushed me into the arms of Tufte changed everything. And still more credit belongs to all the people who helped gather obscure bits of data from all parts of the business, from my colleagues in accounting to those in every department in the company, many of whom built their own models for their own areas, maintaining and iterating them with a regular cadence because they knew every month I'd come knocking and asking questions.

I'm convinced that because Joy knew every part of our business as well or better than almost anyone running them, she was one of those rare CFO's that can play offense in addition to defense. Almost every other CFO I've met hews close to the stereotype; always reigning in spending, urging more fiscal conservatism, casting a skeptical eye on any bold financial transactions. Joy could do that better than the next CFO, but when appropriate she would urge us to spend more with a zeal that matched Jeff's. She, like many visionary CEO's, knew that sometimes the best defense is offense, especially when it comes to internet markets, with their pockets of winner-take-all contests, first mover advantages, and network effects.

It still surprises me how many companies don't help their employees understand the numeric workings of their business. One goes through orientation and hears about culture, travel policies, where the supply cabinet is, maybe some discussion of mission statements. All valuable, of course. But when was the last time any orientation featured any graphs on the business? Is it that we don't trust the numeracy of our employees? Do we fear that level of radical transparency will overwhelm them? Or perhaps it's a mechanism of control, a sort of "don't worry your little mind about the numbers" and just focus on your piece of the puzzle?

Knowing the numbers isn't enough in and of itself, but as books like Moneyball make clear, doing so can reveal hidden truths, unknown vectors of value (for example, in the case of Billy Beane and the Oakland A's, on base percentage). To this day, people still commonly talk about Amazon not being able to turn a profit for so many years as if it is some Ponzi scheme. Late one night in 1997, a few days after I had started, and about my third or fourth time reading the most recent edition of the Analytics Package cover to back, I knew our hidden truth: all the naysaying about Amazon's profitless business model was a lie. Every dollar of our profit we didn't reinvest into the business, and every dollar we didn't raise from investors to add to that investment, would be just kneecapping ourselves. The only governor of our potential was the breadth of our ambition.

***

What does this have to do with line graphs? A month or two into my job, my manager sent me to a seminar that passed through Seattle. It was a full day course centered around the wisdom in one book, taught by the author. The book was The Visual Display of Quantitative Information, a cult bestseller on Amazon.com, the type of long tail book that, in the age before Amazon, might have remained some niche reference book, and the author was Edward Tufte. It's difficult to conjure, on demand, a full list of the most important books I've read, but this is one.

My manager sent me to the seminar so I could apply the principles of that book to the charts in the Analytics Package. My copy of the book sits on my shelf at home, and it's the book I recommend most to work colleagues.

In contrast to this post, which has buried the lede so far you may never find it, Tufte's book opens with a concise summary of its key principles.

Excellence in statistical graphics consists of complex ideas communicated with clarity, precision, and efficiency. Graphics displays should

  • show the data
  • induce the viewer to think about the substance rather than about methodology, graphic design, the technology of graphic production, or something else
  • avoid distorting what the data have to say
  • present many numbers in a small space
  • make large data sets coherent
  • encourage the eye to compare different pieces of data
  • reveal the data at several levels of detail, from a broad overview to the fine structure
  • serve a reasonably clear purpose: description, exploration, tabulation, or decoration
  • be closely integrated with the statistical and verbal descriptions of a data set.

Graphics reveal data. Indeed graphics can be more precise and revealing than conventional statistical computations.

That's it. The rest of the book is just one beautiful elaboration after another of those first principles. The world in one page.

Of all the graphs, the line graph is the greatest. Of its many forms, the most iconic form, the one I used the most in the Analytics Package, has time as the x-axis and the dimension to be measured as the y-axis. Data trended across time.

One data point is one data point. Two data points, trended across time, tell a story. [I'm joking, please don't tell a story using just two data points] The line graph tells us where we've been, and it points to where things are going. In contemplating why the line points up or down, or why it is flat, one grapples with the fundamental mechanism of what's on study.

It wasn't until I'd produced the Analytics Package graphs for several months that my manager granted me the responsibility of writing the cover letter. It was a momentous day, but the actual task of writing the summary of the state of the business wasn't hard. By looking at each graph and investigating why each had changed in which way from last month to this, I had all the key points worth writing up. Building the graphs was more than half the battle.

So many of the principles in Tufte's book made their way into the Analytics Package. For example, where relevant, each page showed a series of small multiples, with the same scale on X and Y axes, back in the age before small multiples were a thing in spreadsheet programs.

Nowhere was Tufte's influence more felt than in our line graphs. How good can a line graph be? After all, in its components, a line graph is really simple. That's a strength, not a weakness. The advice here is simple, so simple, in fact, one might think all of it is common practice already. It isn't. When I see line graphs shared online, even those from some of the smartest people I follow, almost all of them adhere to very little of what I'm going to counsel.

Perhaps Tufte isn't well read enough, his idea not taught in institutions like business schools that require their students to use Excel. That is all true, but I prefer a simpler explanation: users are lazy, the Excel line graph defaults are poor, and Excel is the most popular charting tool on the planet.

By way of illustration, let's take a data set, build a line graph in Excel, and walk through some of what I had to do when making the Analytics Package each month.

I couldn't find the raw data behind most charts shared online, and I didn't want to use any proprietary data. My friend Dan Wang pointed me at the Google Public Data Explorer, a lot of which seems to built off the World Bank Data Catalog, from which I pulled some raw data, just to save me the time of making up figures.

I used health expenditure per capita (current US$). I picked eight countries and used the data for the full range of years available, spanning 1995-2014. I chose a mix of countries I've visited or lived in, plus some people have spoken to me about in reference to their healthcare systems, but the important point here is that limiting the data series on a line graph matters if the graph is going to be readable. How many data series depends on what you want to study and how closely the lines cluster, how large the spread is. Sometimes it's hard to anticipate until you produce the graph, but suffice it to say that generating a graph just to make one is silly if the result is illegible.

Here's what the latest version of Excel on my Mac produced when I pressed the line graph button after highlighting the data (oddly enough, I found a Recommended Charts dropdown button and the three graphs it recommended were three bar graphs of a variety of forms, definitely not the right choice here, among many places where Excel's default logic is poor). I didn't do anything to this graph, just saved it directly as is, at the exact size and formatting Excel selected.

Not great. Applying Richard Thaler and Cass Sunstein's philosophy from Nudge, if we just improved the defaults in Excel and Powerpoint, the graphic excellence the world over would improve by leaps and bounds. If someone out there works on the charting features in Excel and Powerpoint, hear my cries! The power to elevate your common man is in your hands. Please read Tufte.

As an aside, after the Tufte seminar, I walked up to him and asked what software he used for the graphics in his book. His response? Adobe Illustrator. To produce the results he wanted, he, and presumably his assistants, laid out every pixel by hand. Not that helpful for me in producing the Analytics Package monthly on top of my other job duties, but a comment on the charting quality in Excel that rings true even today.

Let's take my chart above and start editing it for the better, as I did back in my Analytics Package days. Let's start with some obvious problems:

  • The legend is nearly the same height as the graph itself
  • A lot of the lines are really close to each other
  • The figures in the left column could be made more readable with thousands comma separators
  • The chart needs a title

I expanded the graph inside the worksheet to make it easier to see, it was the size of about four postage stamps in the sheet for some reason, and fixed the problems above. Here's that modified version.

Excel should add comma separators for thousands by default. The graph is somewhat better, but the labels are still really small, even if you click and expand the photo above to full size. In addition to adjusting the scale of labels and title, however, what else can we do to improve matters?

I began this post just wanting to share the following simple point, the easiest way to upgrade your Excel line graph:

Remove the legend.

That alone will make your line graph so much better that if it's the only thing you remember for the rest of your life, a generation of audiences will thank you.

The problem with a legend is that it asks the user to bounce their eyes back and forth from the graph to the legend, over and over, trying to hold what is usually some color coding system in their short-term memory.

Look at the chart above. Every time I have to see which line is which country, I have look down to the legend and then back to the graph. If I decide to compare any two data series, I have to look back down and memorize two colors, then look back at the chart. Forget even trying to do it for three countries, or for all of them, which is the whole point of the line graph. In forcing the viewer to interpret your legend, your line graph has already dampened much of its explanatory efficiency.

If you have just two data series, a legend isn't egregious, but it is still inferior to removing the legend. Of course, removing the legend isn't enough.

Remove the legend and label the data series directly on the plot.

Unfortunately, here is where your work gets harder, because, much to my disbelief, there is no option in Excel to label data series in place in a line graph. The only automated option is to use the legend.

If I am wrong, and I would love nothing more than to be wrong, please let me know, but I tried going through every one of Excel's various chart menus, which can be brought up by right clicking on different hot spots on the chart, and couldn't find this option. That Excel forces you to right click on so many obscure hot spots just to bring up various options is bad enough. That you can't find this option at all among the dozens of other useless options is a travesty.

The only fix is to create data series labels by hand. You can find an Insert Text Box option somewhere in the Excel menus and ribbons and bars, so I'll make one for each data series and put them roughly in place so you know which data series is which. Next, the moment of truth.

Select the legend. And delete it.

Undo, then delete it again, just to feel the rush as your chart now expands in size to fill the white space left behind. Feels good.

Next, shrink the actual plot area of the graph by selecting it and opening up some margin on the right side of the graph for placing your labels if there isn't enough. Since people read time series left to right, and since the most recent data is on the far right, you'll want your labels to be there, where the viewers' eyes will flow naturally.

Don't move the labels into exact position just yet. First adjust the sizing of the data labels of the axes and the scale of the graph first. Unfortunately, since these text boxes are floating and not attached to the data series, every time the scale of your chart changes, you have to reposition all the data series labels by hand. So do that last.

I have not used this latest version of Excel before, the charting options seem even more complex than before. To change the format of the labels on the x and y-axis, you right click each axis and select Format Axis. I changed the y-axis text format to currency. But to change the size of the labels on each axis, you have to right click each and then select Font. That those are in separate menus is part of the Excel experience.

In expanding the font size of the x-axis, I decided it was too crowded so I went with every other year. I left aligned the data series labels and tried to position them as precisely as possible by eye. I seem to remember Excel used to allow selecting text boxes and moving them one pixel at a time with the arrow keys, but it didn't work for me, so you may have to find the object alignment dropdown somewhere and select align left for all your labels.

Here's the next iteration of the chart.

You can click on it to see it larger. Already, we're better off than the Excel auto-generated chart by quite a margin. If this were the default, I'd be fairly happy. But there's room for improvement.

The use of color can be helpful, especially with lines that are closely stacked, but what about the color blind? If we were to stick with the coloring scheme, I might change the data series labels to match the color of each line. Again, since the labels are added by hand, you'd have to manually change each label by hand to match the color scheme Excel had selected, and again, it wouldn't fix the issue for color blind viewers. [I didn't have the patience to do this for illustrative purposes, but you can see how matching the coloring of labels to the lines helps if you view this data in Google Data Explorer.]

In The Visual Display of Quantitative Information, Tufte uses very little color. When producing the Analytics Package, I was working with black and white printers and copy machines. Color was a no go, even if it provides an added dimension for your graphics, as for elevation on maps.

While color has the advantage of making it easier to distinguish between two lines which are close to each other, it introduces all sorts of mental associations that are difficult to anticipate and which may just be a distraction. When making a chart like, say, one of the U.S. Presidential Election, using blue for Democrats and red for Republicans is a good idea since the color scheme is widely agreed upon. When distinguishing between departments in your company, or product lines, arbitrary color choices can be noise, or worse, a source of contention (NSFW language warning).

The safer alternative is to use different line styles, regardless of whether your final deliverable is capable of displaying color. Depending on how many data series you have to chart, that may or may not be an option. I looked at the data series line format options, which are labeled Dash Type in this version of Excel, and found a total of eight options, or just enough for my example chart. It takes some work to assign options for maximum legibility; you should which country receives which style based on maximum contrast between lines that cluster.

After a random pass at that, the monochrome line graph looked like this.

No issues for color blind users, but we're stretching the limits of line styles past where I'm comfortable. To me, it's somewhat easier with the colored lines above to trace different countries across time versus each other, though this monochrome version isn't terrible. Still, this chart reminds me, in many ways, of the monochromatic look of my old Amazon Analytics Package, though it is missing data labels (wouldn't fit here) and has horizontal gridlines (mine never did).

We're running into some of these tradeoffs because of the sheer number of data series in play. Eight is not just enough, it is probably too many. Past some number of data series, it's often easier and cleaner to display these as a series of small multiples. It all depends on the goal and what you're trying to communicate.

At some point, no set of principles is one size fits all, and as the communicator you have to make some subjective judgments. For example, at Amazon, I knew that Joy wanted to see the data values marked on the graph, whenever they could be displayed. She was that detail-oriented. Once I included data values, gridlines were repetitive, and y-axis labels could be reduced in number as well.

Tufte advocates reducing non-data-ink, within reason, and gridlines are often just that. In some cases, if data values aren't possible to fit onto a line graph, I sometimes include gridlines to allow for easy calculation of the relative ratio of one value to another (simply count gridlines between the values), but that's an edge case.

For sharp changes, like an anomalous reversal in the slope of a line graph, I often inserted a note directly on the graph, to anticipate and head off any viewer questions. For example, in the graph above, if fewer data series were included, but Greece remained, one might wish to explain the decline in health expenditures starting in 2008 by adding a note in the plot area near that data point, noting the beginning of the Greek financial crisis (I don't know if that's the actual cause, but whatever the reason or theory, I'd place it there).

If we had company targets for a specific metric, I'd note those on the chart(s) in question as a labeled asymptote. You can never remind people of goals often enough.

Just as an example, here's another version of that chart, with fewer data series, data labels, no gridlines, fewer y-axis labels. Also, since the lines aren't clustered together, we no longer need different line styles adding visual noise.

At that size, the data values aren't really readable, but if I were making a chart for Joy or Jeff, I'd definitely add the labels because I knew they'd want that level of detail. At Amazon, also, I typically limited our charts to rolling four or eight quarters, so we'd never have this many data points as on the graph above. Again, at some point you have to determine your audience and your goals and modify your chart to match.

Like a movie, work on a chart is a continuous process. I could generate a couple more iterations on the chart above for different purposes, but you get the idea. At some point you have to print it. Just as you'd add the end credits to a film, the last touch here would be to put a source for the data below the graph, so people can follow up on the raw data themselves.

Before I set off on this exercise, I didn't know much about health care expenditures per capita around the world, except that the United States is the world leader by a wide margin. The graph reveals that, and by what magnitude. Look at China by comparison. What explains China's low expenditures? I might hypothesize a number of reasons, including obvious ones like the huge population there, but it would take further investigation, and perhaps more charts. One reason the Analytics Package grew in time was that some charts beget further charts.

Why did Greece's expenditures per capita go into decline starting in 2008. Was it the financial crisis? Why has Japan reversed its upward trajectory starting in 2012? Should we include some other countries for comparison, and how might we choose the most illuminating set?

Every month that first year at Amazon, I'd spend most my waking hours gathering figures and confirming their accuracy, producing these graphs, and then puzzling over the stories behind their contours. The process of making line graphs was prelude to understanding.

To accelerate that understanding, upgrade your line graphs to be efficient and truthful. Some broadly applicable principles should guide you to the right neighborhood. To recap:

  • Don't include a legend; instead, label data series directly in the plot area. Usually labels to the right of the most recent data point are best. Some people argue that a legend is okay if you have more than one data series. My belief is that they're never needed on any well-constructed line graph.
  • Use thousands comma separators to make large figures easier to read
  • Related to that, never include more precision than is needed in data labels. For example, Excel often chooses two decimal places for currency formats, but most line graphs don't need that, and often you can round to 000's or millions to reduce data label size. If you're measuring figures in the billions and trillions, we don't need to see all those zeroes, in fact it makes it harder to read.
  • Format axis labels to match the format of the figures being measured; if it's US dollars, for example, format the labels as currency.
  • Look at the spacing of axis labels and increase the interval if they are too crowded. As Tufte counsels, always reduce non-data-ink as much as possible without losing communicative power.
  • Start your y-axis at zero (assuming you don't have negative values)
  • Try not to have too many data series; five to eight seems the usual limit, depending on how closely the lines cluster. On rare occasion, it's fine to exceed this; sometimes the sheer volume of data series is the point, to show a bunch of lines clustered. These are edge cases for a reason, however.
  • If you have too many data series, consider using small multiples if the situation warrants, for example if the y-axes can match in scale across all the multiples.
  • Respect color blind users and those who may not be able to see your charts with color, for example on a black and white printout, and have options for distinguishing data series beyond color, like line styles. At Amazon, as I dealt with so many figures, I always formatted negative numbers to be red and enclosed in parentheses for those who wouldn't see the figures in color.
  • Include explanations for anomalous events directly on the graph; you may not always be there in person to explain your chart if it travels to other audiences.
  • Always note, usually below the graph, the source for the data.

Some other suggestions which are sometimes applicable:

  • Display actual data values on the graph if people are just going to ask what the figures are anyway, and if they fit cleanly. If you include data labels, gridlines may not be needed. In fact, they may not be needed even if you don't include data labels.
  • Include targets for figures as asymptotes to help audiences see if you're on track to reach them.

Why is The Visual Display of Quantitative Information such a formative text in my life? If it were merely a groundbreaking book on graphic excellence, it would remain one of my trusted references, sitting next to Garner's Modern American Usage, always within arm's reach. It wouldn't be a book I would push on those who never make graphs and charts.

The reason the book influenced me so deeply is that it is actually a book about the pursuit of truth through knowledge. It is ostensibly about producing better charts; what stays with you is the principles for general clarity of thought. Reading the book, chiseling away at my line graphs late nights, talking to people all over the company to understand what might explain each of them, gave me a path towards explaining the past and predicting the future. Ask anyone about any work of art they love, whether it's a book or a movie or an album, and it's never just about what it's about. I haven't read Zen and the Art of Motorcycle Maintenance; I'm guessing it wasn't written just for motorcycle enthusiasts.

A good line graph is a fusion of right and left brain, of literacy and numeracy. Just numbers alone aren't enough to explain the truth, but accurate numbers, represented truthfully, are a check on our anecdotal excesses, confirmation biases, tribal affiliations.

I'm reminded of Tufte's book whenever I brush against tendrils of many movements experiencing a moment online: rationalism, the Nate Silver/538 school of statistics-backed journalism, infographics, UX/UI/graphic design, pop economics, big history. And, much to my dismay, I'm reminded of the book most every time I see a line graph that could use some visual editing. Most people are lazy, most people use the defaults, and the defaults of the most popular charting application on the planet, Excel, are poor.

[Some out there may ask about Apple's Numbers. I tried it a bit, and while it's aesthetically cleaner than Excel, it's such a weak spreadsheet overall that I couldn't make the switch. I dropped Powerpoint for Keynote, though both have some advantages. Neither, unfortunately, includes a great charting tool, though they are simpler in function than the one in Excel. Google Sheets is, like Numbers, a really weak spreadsheet, and it's just plain ugly. If someone out there knows of a superior charting tool, one that doesn't require making charts in Illustrator like Tufte does, please let me know.] 

I love this exchange early on in Batman Begins between Liam Neeson R'as Al Ghul (though he was undercover as Henri Ducard at the time) and Christian Bale's Bruce Wayne.

Bruce Wayne: You're vigilantes.
 
Henri Ducard: No, no, no. A vigilante is just a man lost in the scramble for his own gratification. He can be destroyed, or locked up. But if you make yourself more than just a man, if you devote yourself to an ideal, and if they can't stop you, then you become something else entirely.
 
Bruce Wayne: Which is?
 
Henri Ducard: Legend, Mr. Wayne.
 

It is absurdly self-serious in the way that nerds love their icons to be treated by mainstream pop culture, and I love it for its broad applicability. I've been known to drop some version of it in meetings all the time, my own Rickroll, but no one seems to find it amusing.

In this case, the passage needs some tweaking. But please do still read it with Liam Neeson's trademark gravitas.

A line graph is just another ugly chart lost in the scramble for its own gratification in a slide deck no one wants to read. It can be disregarded, forgotten. But if you make your graph more than just the default Excel format, if you devote yourself to Tufte's ideals, then your graph becomes something else entirely.

Which is?

A line graph without a legend. Remove the legend, Mr. Wayne, and become a legend.

Revisiting The Odyssey

What a pleasant surprise, to have something wonderful that you hadn't heard of just drop out of the sky one day, like Beyonce's Lemonade. That's how I felt about Emily Wilson's new translation of The Odyssey, which arrived on my Kindle yesterday (a physical copy isn't due on my doorstep until next week; this is the rare book for which I wanted one of each format).

So far, I've read more about Wilson's thought process behind her translation than the actual translation itself, but even that is delightful:

In planning to translate the poem into English, my first thoughts were of style. The original is written in a highly rhythmical form of verse. It reads nothing like prose and nothing like any spoken or nonpoetic kinds of discourse. Many modern poets in the Anglo-American tradition write free verse, and modern British and American readers are not usually accustomed to reading long narratives with a regular metrical beat, except for earlier literature like Shakespeare. Most contemporary translators of Homer have not attempted to create anything like a regular line beat, though they often lay out their text as if it were verse. But The Odyssey is a poem, and it needs to have a predictable and distinctive rhythm that can be easily heard when the text is read out loud. The original is in six-footed lines (dactylic hexameters), the conventional meter for archaic Greek narrative verse. I used iambic pentameter, because it is the conventional meter for regular English narrative verse—the rhythm of Chaucer, Shakespeare, Milton, Byron, Keats, and plenty of more recent anglophone poets. I have spent many hours reading aloud, both the Greek original and my own work in progress. Homer's music is quite different from mine, but my translation sings to its own regular and distinctive beat. 
 
My version is the same length as the original, with exactly the same number of lines. I chose to write within this difficult constraint because any translation without such limitations will tend to be longer than the original, and I wanted a narrative pace that could match its stride to Homer's nimble gallop. Moreover, in reading the original, one is constantly aware of the rhythms and the units that make up elements of every line, as well as of the ongoing movement of the narrative—like a large, elaborate piece of embroidery made of tiny, still visible stitches. I wanted my translation to mark its own nature as a web of poetic language, with a sentence structure that is, like that of Homer, audibly built up out of smaller units of sense. There is often a notion, especially in the Anglo-American world, that a translation is good insofar as it disguises its own existence as a translation; translations are praised for being "natural." I hope that my translation is readable and fluent, but that its literary artifice is clearly apparent. 
 
Matthew Arnold famously claimed that translators of Homer must convey four supposedly essential qualities of Homeric style: plainness, simplicity, directness of thought, and nobility. But Homeric style is actually quite often redundant and very often repetitious—not particularly simple or direct. Homer is also very often not "noble": the language is not colloquial, and it avoids obscenity, but it is not bombastic or grandiloquent. The notion that Homeric epic must be rendered in grand, ornate, rhetorically elevated English has been with us since the time of Alexander Pope. It is past time, I believe, to reject this assumption. Homer's language is markedly rhythmical, but it is not difficult or ostentatious, The Odyssey relies on coordinated, not subordinated syntax ("and then this, and then this, and then this," rather than "although this, because of that, when this, which was this, on account of that"). I have frequently aimed for a certain level of simplicity, often using fairly ordinary, straightforward, and readable English. In using language that is largely simple, my goal is not to make Homer sound "primitive," but to mark the fact that stylistic pomposity is entirely un-Homeric. I also hope to invite readers to respond more actively with the text. Impressive displays of rhetoric and linguistic force are a good way to seem important and invite a particular kind of admiration, but they tend to silence dissent and discourage deeper modes of engagement. A consistently elevated style can make it harder for readers to keep track of what is at stake in the story. My translation is, I hope, recognizable as an epic poem, but it is one that avoids trumpeting its own status with bright, noisy linguistic fireworks, in order to invite a more thoughtful consideration of what the narrative means, and the ways it matters.
 

I'm with her on all of that. Iambic pentameter! Be still my bleeding heart.

Nodding along when she makes choices like this:

The formulaic elements in Homer, especially the repeated epithets, pose a particular challenge. The epithets applied to Dawn, Athena, Hermes, Zeus, Penelope, Telemachus, Odysseus, and the suitors repeat over and over in the original. But in my version, I have chosen deliberately to interpret these epithets in several different ways, depending on the demands of the scene at hand. I do not want to deceive the unsuspecting reader about the nature of the original poem; rather, I hope to be truthful about my own text—its relationships with its readers and with the original. In an oral or semiliterate culture, repeated epithets give a listener an anchor in a quick-moving story. In a highly literate society such as our own, repetitions are likely to feel like moments to skip. They can be a mark of writerly laziness or unwillingness to acknowledge one's own interpretative position, and can send a reader to sleep. I have used the opportunity offered by the repetitions to explore the multiple different connotations of each epithet.
 

I try not to spend too much time fetishizing craft, but when it comes to translation, it is inseparable from the thing.

This past Sunday's NYTimes Magazine included a feature on Wilson's accomplishment. In this age where we celebrate women breaking through in fields previously occupied by only white men, being the first woman to translate one of the great works of Western literature resounds at many levels.

The NYTimes feature spends some time laying bare the translator's hand. Take, for example, the thought that went into the opening line of the epic itself, and how varied its forms in all the translations that had been come before Wilson's. Just one line and already so many forks.

The first of these changes is in the very first line. You might be inclined to suppose that, over the course of nearly half a millennium, we must have reached a consensus on the English equivalent for an old Greek word, polytropos. But to consult Wilson’s 60 some predecessors, living and dead, is to find that consensus has been hard to come by. Chapman starts things off, in his version, with “many a way/Wound with his wisdom”; John Ogilby counters with the terser “prudent”; Thomas Hobbes evades the word, just calling Odysseus “the man.” Quite a range, and we’ve barely started. There’s Alexander Pope’s “for wisdom’s various arts renown’d”; William Cowper’s “For shrewdness famed/And genius versatile”; H.F. Cary’s “crafty”; William Sotheby’s “by long experience tried”; Theodore Buckley’s “full of resources”; Henry Alford’s “much-versed”; Philip Worsley’s “that hero”; the Rev. John Giles’s “of many fortunes”; T.S. Norgate’s “of many a turn”; George Musgrave’s “tost to and fro by fate”; the Rev. Lovelace Bigge-Wither’s “many-sided-man”; George Edgington’s “deep”; William Cullen Bryant’s “sagacious”; Roscoe Mongan’s “skilled in expedients”; Samuel Henry Butcher and Andrew Lang’s “so ready at need”; Arthur Way’s “of craft-renown”; George Palmer’s “adventurous”; William Morris’s “shifty”; Samuel Butler’s “ingenious”; Henry Cotterill’s “so wary and wise”; Augustus Murray’s “of many devices”; Francis Caulfeild’s “restless”; Robert Hiller’s “clever”; Herbert Bates’s “of many changes”; T.E. Lawrence’s “various-minded”; William Henry Denham Rouse’s “never at a loss”; Richmond Lattimore’s “of many ways”; Robert Fitzgerald’s “skilled in all ways of contending”; Albert Cook’s “of many turns”; Walter Shewring’s “of wide-ranging spirit”; Allen Mandelbaum’s “of many wiles”; Robert Fagles’s “of twists and turns”; all the way to Stanley Lombardo’s “cunning.”
 
Of the 60 or so answers to the polytropos question to date, the 36 given above couldn’t be less uniform (the two dozen I omit repeat, with minor variations, earlier solutions); what unites them is that their translators largely ignore the ambiguity built into the word they’re translating. Most opt for straightforward assertions of Odysseus’s nature, descriptions running from the positive (crafty, sagacious, versatile) to the negative (shifty, restless, cunning). Only Norgate (“of many a turn”) and Cook (“of many turns”) preserve the Greek roots as Wilson describes them — poly (“many”), tropos (“turn”) — answers that, if you produced them as a student of classics, much of whose education is spent translating Greek and Latin and being marked correct or incorrect based on your knowledge of the dictionary definitions, would earn you an A. But to the modern English reader who does not know Greek, does “a man of many turns” suggest the doubleness of the original word — a man who is either supremely in control of his life or who has lost control of it? Of the existing translations, it seems to me that none get across to a reader without Greek the open question that, in fact, is the opening question of the “Odyssey,” one embedded in the fifth word in its first line: What sort of man is Odysseus?
 

All that variation in just one word. Let's telescope out to the entire opening paragraph or stanza (the Paris Review published an excerpt from the opening of Wilson's translation), the invocation of the Muse:

Tell me about a complicated man.
Muse, tell me how he wandered and was lost
when he had wrecked the holy town of Troy,
and where he went, and who he met, the pain
he suffered in the storms at sea, and how
he worked to save his life and bring his men
back home. He failed to keep them safe; poor fools,
they ate the Sun God’s cattle, and the god
kept them from home. Now goddess, child of Zeus,
tell the old story for our modern times.
Find the beginning.
 

Here is how another popular translation, by Robert Fagles, handles the same passage:

Sing to me of the man, Muse, the man of twists and turns ...
driven time and again off course, once he had plundered
the hallowed heights of Troy.
Many cities of men he saw and learned their minds,
many pains he suffered, heartsick on the open sea,
fighting to save his life and bring his comrades home.
But he could not save them from disaster, hard as he strove—
the recklessness of their own ways destroyed them all,
the blind fools, they devoured the cattle of the Sun
and the Sungod blotted out the day of their return.
Launch out on his story, Muse, daughter of Zeus,
start from where you will—sing for our time too.
 

Here is Richmond Lattimore's opening. I can't remember if I had to read this or Fagles' in high school or college, it was one or the other.

Tell me, Muse, of the man of many ways, who was driven
far journeys, after he had sacked Troy’s sacred citadel.
Many were they whose cities he saw, whose minds he learned of,
many the pains he suffered in his spirit on the wide sea,
struggling for his own life and the homecoming of his companions.
Even so he could not save his companions, hard though
he strove to; they were destroyed by their own wild recklessness,
fools, who devoured the oxen of Helios, the Sun God,
and he took away the day of their homecoming.
From some point here, goddess, daughter of Zeus,
speak, and begin our story.
 

While they all have their virtues, it's impossible to ignore the startling directness of Wilson's version. It is direct, more concise, and has a lyrical momentum from the iambic pentameter that adds to its muscularity. "He failed to keep them safe" is stronger in tone than "he could not save them from disaster, hard as he strove" or "he could not save his companions, hard though he strove to." Wilson may choose to leave out the striving because, in the previous sentence, as all the translations include, it is already noted that Odysseus suffered many pains to save his life and bring his men home. How hard he strove is somewhat repetitive, so Wilson just nixes it.

"Complicated man" is about as tidy a way to characterize a character who contains multitudes. Fagles' "the man of twists and turns" doesn't register much to me except the image of a pretzel. Lattimore's "the man of many ways" is intriguing, less precise than Wilson's "complicated man" but hinting at both Odysseus' resourcefulness and complexity.

Direct does not mean Wilson dispenses with fun. Later in the opening we have this:

So why do you dismiss Odysseus?”
 

Let your tongue tap over that like a rock skip-skipping o'er a pond.

I could not help flipping ahead to what I consider one of the most iconic scenes in Western literature, in which Odysseus and his men, having blinded the one-eyed monster Polyphemus, are in a boat, escaping the island where the creature had held them hostage. Polyphemus does not know where they are, he is beside himself with pain and fury. Just as Odysseus and his men are about to escape unharmed, he turns back to face his vanquished foe. He cannot help himself.

When I had gone as far as shouts can carry,
I jeered back,

‘Hey, you, Cyclops! Idiot!
The crew trapped in your cave did not belong
to some poor weakling. Well, you had it coming!
You had no shame at eating your own guests!
So Zeus and other gods have paid you back.’

My taunting made him angrier. He ripped
a rock out of the hill and hurled it at us.
It landed right in front of our dark prow,
and almost crushed the tip of the steering oar.
The stone sank in the water; waves surged up.
The backflow all at once propelled the ship
landwards; the swollen water pushed us with it.
I grabbed a big long pole, and shoved us off.
I told my men, ‘Row fast, to save your lives!’
and gestured with my head to make them hurry.
They bent down to their oars and started rowing.
We got out twice as far across the sea,
and then I called to him again. My crew
begged me to stop, and pleaded with me.

‘Please!
Calm down! Why are you being so insistent
and taunting this wild man? He hurled that stone
and drove our ship right back to land. We thought
that we were going to die. If he had heard us,
he would have hurled a jagged rock and crushed
our heads and wooden ship. He throws so hard!

But my tough heart was not convinced; I was
still furious, and shouted back again,

‘Cyclops! If any mortal asks you how
your eye was mutilated and made blind,
say that Odysseus, the city-sacker,
Laertes’ son, who lives in Ithaca,
destroyed your sight.’ 
 

This is, for my money, one of the seminal moments in Western literature. It's the birth of ritualized trash talk and boasting, the defining instance of taunting in the Western canon. Every time you see a basketball player get all up in the mug of some opponent after dunking on them, every time Cam Newton pantomines opening his jersey to reveal the Superman cape, every time a rapper refers to himself in the third person after performing lyrical violence on a nemesis, every time Roy Jones Jr. gave a shout out to Pensacola after one of his boxing victories, it all traces back to this moment when Odysseus can't help claiming personal credit for having outwitted the beast, giving himself a title (city-sacker) and naming himself in relation to his family (Laertes' son) and his home (Ithaca).

When Danaerys Targaryen on Game of Thrones introduces herself as "Daenerys Stormborn of the House Targaryen, First of Her Name, the Unburnt, Queen of the Andals and the First Men, Khaleesi of the Great Grass Sea, Breaker of Chains, and Mother of Dragons," she should credit Odysseus,, the city-sacker, Laertes' son, who lives in Ithaca. When we use the term "making our name," we call back to Odysseus, who in that moment established the now familiar tradition of referring to oneself in the third person.

It comes with a cost. His pride and arrogance not only endanger his men by revealing their location and allowing Polyphemus to better target his next rock throw, but in making his name, Odysseus gives Polyphemus a specific target. As anyone who has read mythology or fairy tales knows, a specific name is needed to target curses from afar. Polyphemus' father just happens to be Poseidon, a god, and it's to poppa that he turns for help.

But he prayed
holding his arms towards the starry sky,
‘Listen, Earth-Shaker, Blue-Haired Lord Poseidon:
acknowledge me your son, and be my father.
Grant that Odysseus, the city-sacker,
will never go back home. Or if it is
fated that he will see his family,
then let him get there late and with no honor,
in pain and lacking ships, and having caused
the death of all his men, and let him find
more trouble in his own house.’

Blue Poseidon
granted his son’s prayer.
 

And so Odysseus brings a curse upon himself, his family, and his men. All of the above comes true, as prophecies are wont to do in stories like this.

I'd be inclined to chide him for his hubris, but wouldn't the internet be better today if trolls didn't hide like cowards behind the veil of anonymity? Face your critics, and name yourself, anonymous neo-Nazis. Rereading this passage, I couldn't help but think of Peter Cvjetanovic, the student who marched in the Charlottesville protests and was identified in a photo.

I am Peter Cvjetanovic,
Charlottesville city sacker,
neo-Nazi sympathizer
University Nevada Reno student
and campus escort service driver,
or used to be.

Let's not end there. Let's leave with a more pleasant example, when Russell Crowe removes his mask in perhaps the most iconic arena of battle in Western myth, the Colosseum, and names himself, as his ancestor Odysseus once did. If you're looking for inspiration for the next draft of your Twitter bio, now you have it, all you tweeter of tweets.

10 browser tabs

1. Love in the Time of Robots

“Is it difficult to play with her?” the father asks. His daughter looks to him, then back at the android. Its mouth begins to open and close slightly, like a dying fish. He laughs. “Is she eating something?”
 
The girl does not respond. She is patient and obedient and listens closely. But something inside is telling her to resist. 
 
“Do you feel strange?” her father asks. Even he must admit that the robot is not entirely believable.
 
Eventually, after a few long minutes, the girl’s breathing grows heavier, and she announces, “I am so tired.” Then she bursts into tears.
 
That night, in a house in the suburbs, her father uploads the footage to his laptop for posterity. His name is Hiroshi Ishi­guro, and he believes this is the first record of a modern-day android.
 

Reads like the treatment for a science fiction film, some mashup of Frankenstein, Pygmalion, and Narcissus. One incredible moment after another, and I'll grab just a few excerpts, but the whole thing is worth reading.

But he now wants something more. Twice he has witnessed others have the opportunity, however confusing, to encounter their robot self, and he covets that experience. Besides, his daughter was too young, and the newscaster, though an adult, was, in his words, merely an “ordinary” person: Neither was able to analyze their android encounter like a trained scientist. A true researcher should have his own double. Flashing back to his previous life as a painter, Ishi­guro thinks: This will be another form of self-portrait. He gives the project his initials: Geminoid HI. His mechanical twin.
 

Warren Ellis, in a recent commencement speech delivered at the University of Essex, said:

Nobody predicted how weird it’s gotten out here.  And I’m a science fiction writer telling you that.  And the other science fiction writers feel the same.  I know some people who specialized in near-future science fiction who’ve just thrown their hands up and gone off to write stories about dragons because nobody can keep up with how quickly everything’s going insane.  It’s always going to feel like being thrown in the deep end, but it’s not always this deep, and I’m sorry for that.
 

The thing is, far future sci-fi is likely to be even more off base now given how humans are evolving in lock step with the technology around them. So we need more near future sci-fi, of a variety smarter than Black Mirror, to grapple with the implications.

Soon his students begin comparing him to the Geminoid—“Oh, professor, you are getting old,” they tease—and Ishi­guro finds little humor in it. A few years later, at 46, he has another cast of his face made, to reflect his aging, producing a second version of HI. But to repeat this process every few years would be costly and hard on his vanity. Instead, Ishi­guro embraces the logi­cal alternative: to alter his human form to match that of his copy. He opts for a range of cosmetic procedures—laser treatments and the injection of his own blood cells into his face. He also begins watching his diet and lifting weights; he loses about 20 pounds. “I decided not to get old anymore,” says Ishi­guro, whose English is excellent but syntactically imperfect. “Always I am getting younger.”
 
Remaining twinned with his creation has become a compulsion. “Android has my identity,” he says. “I need to be identical with my android, otherwise I’m going to lose my identity.” I think back to another photo of his first double’s construction: Its robot skull, exposed, is a sickly yellow plastic shell with openings for glassy teeth and eyeballs. When I ask what he was thinking as he watched this replica of his own head being assembled, Ishi­guro says, perhaps only half-joking, “I thought I might have this kind of skull if I removed my face.”
 
Now he points at me. “Why are you coming here? Because I have created my copy. The work is important; android is important. But you are not interested in myself.”
 

This should be some science fiction film, only I'm not sure who our great science fiction director is. The best examples may be too old to want to look upon such a story as anything other than grotesque and horrific.

2. Something is wrong on the internet by James Bridle

Of course, some of what's on the internet really is grotesque and horrific. 

Someone or something or some combination of people and things is using YouTube to systematically frighten, traumatise, and abuse children, automatically and at scale, and it forces me to question my own beliefs about the internet, at every level. 
 

Given how much my nieces love watching product unwrapping and Peppa the Pig videos on YouTube, this story was induced a sense of dread I haven't felt since the last good horror film I watched, which I can't remember anymore since the world has run a DDOS on my emotions.

We often think of a market operating at peak efficiency as sending information back and forth between supply and demand, allowing the creation of goods that satisfy both parties. In the tech industry, the wink-wink version of that is saying that pornography leads the market for any new technology, solving, as it does, the two problems the internet is said to solve better, at scale, than any medium before it: loneliness and boredom.

Bridle's piece, however, finds the dark cul-de-sacs and infected runaway processes which have branched out from the massive marketplace that is YouTube. I decided to follow a Peppa the Pig video on the service and started tapping on Related Videos, like I imagine one of my nieces doing, and quickly wandered into a dark alleyway where I saw some video which I would not want any of them watching. As Bridle did, I won't link to what I found; suffice to say it won't take you long to stumble on some of it if you want, or perhaps even if you don't.

What's particularly disturbing is the somewhat bizarre, inexplicably grotesque nature of some of these video remixes. David Cronenberg is known for his body horror films; these YouTube videos are like some perverse variant of that, playing with popular children's iconography.

Facebook and now Twitter are taking heat for disseminating fake news, and that is certainly a problem worth debating, but with that problem we're talking about adults. Children don't have the capacity to comprehend what they're seeing, and given my belief in the greater effect of sight, sound, and motion, I am even more disturbed by this phenomenon.

A system where it's free to host videos to a global audience, where this type of trademark infringement weaponizes brand signifiers with seeming impunity, married with increasingly scalable content production and remixes using technology, allows for the type of scalable problem we haven't seen before.

The internet has enabled all types of wonderful things at scale; we should not be surprised that it would foster the opposite. But we can, and should, be shocked.

3. FDA approves first blood sugar monitor without finger pricks

This is exciting. One view which seems to be common wisdom these days when it comes to health is that it's easier to lose weight and impact your health through diet than exercise. But one of the problems of the feedback loop in diet (and exercise, actually) is how slow it is. You sneak a few snacks here and there walking by the company cafeteria every day, and a month later you hop on the scale and emit a bloodcurdling scream as you realize you've gained 8 pounds.

A friend of mine had gestational diabetes during one of her pregnancies and got a home blood glucose monitor. You had to prick your finger and draw blood to get your blood glucose reading, but curious, I tried it before and after a BBQ.

To see what various foods did to my blood sugar in near real-time was a real eye-opener. Imagine in the future when one could see what a few french fries and gummy bears did to your blood sugar, or when the reading could be built into something like an Apple Watch, without having to draw blood each time. I don't mind the sight of blood, but I'd prefer not to turn my finger tips into war zones.

Faster feedback might transform dieting into something more akin to deliberate practice. Given that another popular theory of obesity is that it's an insulin phenomenon, tools like this, built for diabetes, might have much mass market impact.

4.  Ingestable ketones

Ingestable ketones have been a recent sort of holy grail for endurance athletes, and now HVMN is bringing one to market. Ketogenic diets are all the rage right now, but for an endurance athlete, the process of being able to fuel oneself on ketones has always sounded like a long and miserable process.

The body generates ketones from fat when low on carbs or from fasting. The theory is that endurance athletes using ketones rather than glycogen from carbs require less oxygen and thus can work out longer.

I first heard about the possibility of exogenous ketones for athletes from Peter Attia. As he said then, perhaps the hardest thing about ingesting exogenous ketones is the horrible taste, which caused him to gag and nearly vomit in his kitchen. It doesn't sound like the taste problem has been solved.

Until we get the pill that renders exercise obsolete, however, I'm curious to give this a try. If you decide to pre-order, you can use my referral code to get $15 off.

5. We Are Nowhere Close to the Limits of Athletic Performance

By comparison, the potential improvements achievable by doping effort are relatively modest. In weightlifting, for example, Mike Israetel, a professor of exercise science at Temple University, has estimated that doping increases weightlifting scores by about 5 to 10 percent. Compare that to the progression in world record bench press weights: 361 pounds in 1898, 363 pounds in 1916, 500 pounds in 1953, 600 pounds in 1967, 667 pounds in 1984, and 730 pounds in 2015. Doping is enough to win any given competition, but it does not stand up against the long-term trend of improving performance that is driven, in part, by genetic outliers. As the population base of weightlifting competitors has increased, outliers further and further out on the tail of the distribution have appeared, driving up world records.
 
Similarly, Lance Armstrong’s drug-fuelled victory of the 1999 Tour de France gave him a margin of victory over second-place finisher Alex Zulle of 7 minutes, 37 seconds, or about 0.1 percent.3 That pales in comparison to the dramatic secular increase in speeds the Tour has seen over the past half century: Eddy Merckx won the 1971 tour, which was about the same distance as the 1999 tour, in a time 5 percent worse than Zulle’s. Certainly, some of this improvement is due to training methods and better equipment. But much of it is simply due to the sport’s ability to find competitors of ever more exceptional natural ability, further and further out along the tail of what’s possible.
 

In the Olympics, to take the most celebrated athletic competition, victors are celebrated with videos showing them swimming laps, tossing logs in a Siberian tundra, running through a Kenyan desert. We celebrate the work, the training. Good genes are given narrative short shrift. Perhaps we should show a picture of their DNA, just to give credit where much credit is due?

If I live a normal human lifespan, I expect to live to see special sports leagues and divisions created for athletes who've undergone genetic modification in the future. It will be the return of the freak show at the circus, but this time for real. I've sat courtside and seen people like Lebron James, Giannis Antetokounmpo, Kevin Durant, and Joel Embiid walk by me. They are freaks, but genetic engineering might produce someone who stretch our definition of outlier.

In other words, it is highly unlikely that we have come anywhere close to maximum performance among all the 100 billion humans who have ever lived. (A completely random search process might require the production of something like a googol different individuals!)
 
But we should be able to accelerate this search greatly through engineering. After all, the agricultural breeding of animals like chickens and cows, which is a kind of directed selection, has easily produced animals that would have been one in a billion among the wild population. Selective breeding of corn plants for oil content of kernels has moved the population by 30 standard deviations in roughly just 100 generations.6 That feat is comparable to finding a maximal human type for a specific athletic event. But direct editing techniques like CRISPR could get us there even faster, producing Bolts beyond Bolt and Shaqs beyond Shaq.
 

6. Let's set half a percent as the standard for statistical significance

My many-times-over coauthor Dan Benjamin is the lead author on a very interesting short paper "Redefine Statistical Significance." He gathered luminaries from many disciplines to jointly advocate a tightening of the standards for using the words "statistically significant" to results that have less than a half a percent probability of occurring by chance when nothing is really there, rather than all results that—on their face—have less than a 5% probability of occurring by chance. Results with more than a 1/2% probability of occurring by chance could only be called "statistically suggestive" at most. 
 
In my view, this is a marvelous idea. It could (a) help enormously and (b) can really happen. It can really happen because it is at heart a linguistic rule. Even if rigorously enforced, it just means that editors would force people in papers to say "statistically suggestive for a p of a little less than .05, and only allow the phrase "statistically significant" in a paper if the p value is .005 or less. As a well-defined policy, it is nothing more than that. Everything else is general equilibrium effects.
 

Given the replication crisis has me doubting almost every piece of conventional wisdom I've inherited in my life, I'm okay with this.

7. We're surprisingly unaware of when our own beliefs change

If you read an article about a controversial issue, do you think you’d realise if it had changed your beliefs? No one knows your own mind like you do – it seems obvious that you would know if your beliefs had shifted. And yet a new paper in The Quarterly Journal of Experimental Psychology suggests that we actually have very poor “metacognitive awareness” of our own belief change, meaning that we will tend to underestimate how much we’ve been swayed by a convincing article.
 
The researchers Michael Wolfe and Todd Williams at Grand Valley State University said their findings could have implications for the public communication of science. “People may be less willing to meaningfully consider belief inconsistent material if they feel that their beliefs are unlikely to change as a consequence,” they wrote.
 

Beyond being an interesting result, I link to this as an example of a human readable summary of a research paper. This his how this article summarize the research study and its results:

The researchers recruited over two hundred undergrads across two studies and focused on their beliefs about whether the spanking/smacking of kids is an effective form of discipline. The researchers chose this topic deliberately in the hope the students would be mostly unaware of the relevant research literature, and that they would express a varied range of relatively uncommitted initial beliefs.
 
The students reported their initial beliefs about whether spanking is an effective way to discipline a child on a scale from “1” completely disbelieve to “9” completely believe. Several weeks later they were given one of two research-based texts to read: each was several pages long and either presented the arguments and data in favour of spanking or against spanking. After this, the students answered some questions to test their comprehension and memory of the text (these measures varied across the two studies). Then the students again scored their belief in whether spanking is effective or not (using the same 9-point scale as before). Finally, the researchers asked them to recall what their belief had been at the start of the study.
 
The students’ belief about spanking changed when they read a text that argued against their own initial position. Crucially, their memory of their initial belief was shifted in the direction of their new belief – in fact, their memory was closer to their current belief than their original belief. The more their belief had changed, the larger this memory bias tended to be, suggesting the students were relying on their current belief to deduce their initial belief. The memory bias was unrelated to the measures of how well they’d understood or recalled the text, suggesting these factors didn’t play a role in memory of initial belief or awareness of belief change.
 

Compare this link above to the abstract of the paper itself:

When people change beliefs as a result of reading a text, are they aware of these changes? This question was examined for beliefs about spanking as an effective means of discipline. In two experiments, subjects reported beliefs about spanking effectiveness during a prescreening session. In a subsequent experimental session, subjects read a one-sided text that advocated a belief consistent or inconsistent position on the topic. After reading, subjects reported their current beliefs and attempted to recollect their initial beliefs. Subjects reading a belief inconsistent text were more likely to change their beliefs than those who read a belief consistent text. Recollections of initial beliefs tended to be biased in the direction of subjects’ current beliefs. In addition, the relationship between the belief consistency of the text read and accuracy of belief recollections was mediated by belief change. This belief memory bias was independent of on-line text processing and comprehension measures, and indicates poor metacognitive awareness of belief change.
 

That's actually one of the better research abstracts you'll read and still it reflects the general opacity of the average research abstract. I'd argue that some of the most important knowledge in the world is locked behind abstruse abstracts.

Why do researchers write this way? Most tell me that researchers write for other researchers, and incomprehensible prose like this impresses their peers. What a tragedy. As my longtime readers know, I'm a firm believer in the power of the form of a message. We continue to underrate that in all aspects of life, from the corporate world to our personal lives, and here, in academia.

Then again, such poor writing keeps people like Malcolm Gladwell busy transforming such insight into breezy reads in The New Yorker and his bestselling books.

8. Social disappointment explains chimpanzees' behaviour in the inequity aversion task

As an example of the above phenomenon, this paper contains an interesting conclusion, but try to parse this abstract:

Chimpanzees’ refusal of less-preferred food when an experimenter has previously provided preferred food to a conspecific has been taken as evidence for a sense of fairness. Here, we present a novel hypothesis—the social disappointment hypothesis—according to which food refusals express chimpanzees' disappointment in the human experimenter for not rewarding them as well as they could have. We tested this hypothesis using a two-by-two design in which food was either distributed by an experimenter or a machine and with a partner present or absent. We found that chimpanzees were more likely to reject food when it was distributed by an experimenter rather than by a machine and that they were not more likely to do so when a partner was present. These results suggest that chimpanzees’ refusal of less-preferred food stems from social disappointment in the experimenter and not from a sense of fairness.
 

Your average grade school English teacher would slap a failing grade on this butchery of the English language.

9. Metacompetition: Competing Over the Game to be Played

When CDMA-based technologies took off in the US, companies like QualComm that work on that standard prospered; metacompetitions between standards decide the fates of the firms that adopt (or reject) those standards.

When an oil spill raises concerns about the environment, consumers favor businesses with good environmental records; metacompetitions between beliefs determine the criteria we use to evaluate whether a firm is “good.”

If a particular organic foods certification becomes important to consumers, companies with that certification are favored; metacompetitions between certifications determines how the quality of firms is measured.
 
In all these examples, you could be the very best at what you do, but lose in the metacompetition over what criteria will matter. On the other hand, you may win due to a metacompetition that protects you from fierce rivals who play a different game.
 
Great leaders pay attention to metacompetition. They advocate the game they play well, promoting criteria on which they measure up. By contrast, many failed leaders work hard at being the best at what they do, only to throw up their hands in dismay when they are not even allowed to compete. These losers cannot understand why they lost, but they have neglected a fundamental responsibility of leadership. It is not enough to play your game well. In every market in every country, alternative “logics” vie for prominence. Before you can win in competition, you must first win the metacompetition over the game being played.
 

In sports negotiations between owners and players, the owners almost always win the metacompetition game. In the writer's strike in Hollywood in 2007, the writer's guild didn't realize they were losing the metacompetition and thus ended up worse off than before. Amazon surpassed eBay by winning the retail metacompetition (most consumers prefer paying a good, fixed price for a good of some predefined quality than dealing with the multiple axes of complexity of an auction) after first failing at tackling eBay on its direct turf of auctions.

Winning the metacompetition means first being aware of what it is. It's not so easy in a space like, say, social networking, where even some of the winners don't understand what game they're playing.

10. How to be a Stoic

Much of Epictetus’ advice is about not getting angry at slaves. At first, I thought I could skip those parts. But I soon realized that I had the same self-recriminatory and illogical thoughts in my interactions with small-business owners and service professionals. When a cabdriver lied about a route, or a shopkeeper shortchanged me, I felt that it was my fault, for speaking Turkish with an accent, or for being part of an élite. And, if I pretended not to notice these slights, wasn’t I proving that I really was a disengaged, privileged oppressor? Epictetus shook me from these thoughts with this simple exercise: “Starting with things of little value—a bit of spilled oil, a little stolen wine—repeat to yourself: ‘For such a small price, I buy tranquillity.’ ”
 
Born nearly two thousand years before Darwin and Freud, Epictetus seems to have anticipated a way out of their prisons. The sense of doom and delight that is programmed into the human body? It can be overridden by the mind. The eternal war between subconscious desires and the demands of civilization? It can be won. In the nineteen-fifties, the American psychotherapist Albert Ellis came up with an early form of cognitive-behavioral therapy, based largely on Epictetus’ claim that “it is not events that disturb people, it is their judgments concerning them.” If you practice Stoic philosophy long enough, Epictetus says, you stop being mistaken about what’s good even in your dreams.
 

The trendiness of stoicism has been around for quite some time now. I found this tab left over from 2016, and I'm sure Tim Ferriss was espousing it long before then, and not to mention the enduring trend that is Buddhism. That meditation and stoicism are so popular in Silicon Valley may be a measure of the complacency of the region; these seem direct antidotes to the most first world of problems. People everywhere complain of the stresses on their mind from the deluge of information they receive for free from apps on the smartphone with processing power that would put previous supercomputers to shame.

Still, given that stoicism was in vogue in Roman times, it seems to have stood the test of time. Since social media seems to have increased the surface area of our social fabric and our exposure to said fabric, perhaps we could all use a bit more stoicism in our lives. I suspect one reason Curb Your Enthusiasm curdles in the mouth more than before is not just that his rich white man's complaints seem particularly ill timed in the current environment but that he is out of touch with the real nature of most people's psychological stressors now. A guy of his age and wealth probably doesn't spend much time on social media, but if he did, he might realize his grievances no longer match those of the average person in either pettiness or peculiarity.

Chasm of comprehension

Last year, Google's AI AlphaGo beat Korean Lee Sedol in Go, a game many expected humans to continue to dominate for years, if not decades, to come.

With the 37th move in the match’s second game, AlphaGo landed a surprise on the right-hand side of the 19-by-19 board that flummoxed even the world’s best Go players, including Lee Sedol. “That’s a very strange move,” said one commentator, himself a nine dan Go player, the highest rank there is. “I thought it was a mistake,” said the other. Lee Sedol, after leaving the match room, took nearly fifteen minutes to formulate a response. Fan Gui—the three-time European Go champion who played AlphaGo during a closed-door match in October, losing five games to none—reacted with incredulity. But then, drawing on his experience with AlphaGo—he has played the machine time and again in the five months since October—Fan Hui saw the beauty in this rather unusual move
 
Indeed, the move turned the course of the game. AlphaGo went on to win Game Two, and at the post-game press conference, Lee Sedol was in shock. “Yesterday, I was surprised,” he said through an interpreter, referring to his loss in Game One. “But today I am speechless. If you look at the way the game was played, I admit, it was a very clear loss on my part. From the very beginning of the game, there was not a moment in time when I felt that I was leading.”
 

The first time Gary Kasparov sensed deep intelligence in Deep Blue, he described the computer's move as a very human one

I GOT MY FIRST GLIMPSE OF ARTIFICIAL INTELLIGENCE ON Feb. 10, 1996, at 4:45 p.m. EST, when in the first game of my match with Deep Blue, the computer nudged a pawn forward to a square where it could easily be captured. It was a wonderful and extremely human move. If I had been playing White, I might have offered this pawn sacrifice. It fractured Black's pawn structure and opened up the board. Although there did not appear to be a forced line of play that would allow recovery of the pawn, my instincts told me that with so many "loose" Black pawns and a somewhat exposed Black king, White could probably recover the material, with a better overall position to boot.
 
But a computer, I thought, would never make such a move. A computer can't "see" the long-term consequences of structural changes in the position or understand how changes in pawn formations may be good or bad.
 
Humans do this sort of thing all the time. But computers generally calculate each line of play so far as possible within the time allotted. Because chess is a game of virtually limitless possibilities, even a beast like Deep Blue, which can look at more than 100 million positions a second, can go only so deep. When computers reach that point, they evaluate the various resulting positions and select the move leading to the best one. And because computers' primary way of evaluating chess positions is by measuring material superiority, they are notoriously materialistic. If they "understood" the game, they might act differently, but they don't understand.
 
So I was stunned by this pawn sacrifice. What could it mean? I had played a lot of computers but had never experienced anything like this. I could feel--I could smell--a new kind of intelligence across the table. While I played through the rest of the game as best I could, I was lost; it played beautiful, flawless chess the rest of the way and won easily.
 

Later, in the Kasparov-Deep Blue rematch that IBM's computer won, again a move in the 2nd game was pivotal. There is debate or whether the move was a mistake or intentional on the part of the computer, but it flummoxed Kasparov (italics mine):

'I was not in the mood of playing at all,'' he said, adding that after Game 5 on Saturday, he had become so dispirited that he felt the match was already over. Asked why, he said: ''I'm a human being. When I see something that is well beyond my understanding, I'm afraid.''
 
...
 

At the news conference after the game, a dark-eyed and brooding champion said that his problems began after the second game, won by Deep Blue after Mr. Kasparov had resigned what was eventually shown to be a drawn position. Mr. Kasparov said he had missed the draw because the computer had played so brilliantly that he thought it would have obviated the possibility of the draw known as perpetual check.

''I do not understand how the most powerful chess machine in the world could not see simple perpetual check,'' he said. He added he was frustrated by I.B.M.'s resistance to allowing him to see the printouts of the computer's thought processes so he could understand how it made its decisions, and implied again that there was some untoward behavior by the Deep Blue team.

Asked if he was accusing I.B.M. of cheating, he said: ''I have no idea what's happening behind the curtain. Maybe it was an outstanding accomplishment by the computer. But I don't think this machine is unbeatable.''

Mr. Kasparov, who defeated a predecessor of Deep Blue a year ago, won the first game of this year's match, but it was his last triumph, a signal that the computer's pattern of thought had eluded him. He couldn't figure out what its weaknesses were, or if he did, how to exploit them.

Legend has it that a move in Game One and another in Game Two were actually just programming glitches that caused Deep Blue to make random moves that threw Kasparov off, but regardless, the theme is the same: at some point he no longer understood what the program was doing. He no longer had a working mental model, like material advantage, for his computer opponent.

This year, a new version of AlphaGo was unleashed on the world: AlphaGo Zero.

As many will remember, AlphaGo—a program that used machine learning to master Go—decimated world champion Ke Jie earlier this year. Then, the program’s creators at Google’s DeepMind let the program continue to train by playing millions of games against itself. In a paper published in Nature earlier this week, DeepMind revealed that a new version of AlphaGo (which they christened AlphaGo Zero) picked up Go from scratch, without studying any human games at all. AlphaGo Zero took a mere three days to reach the point where it was pitted against an older version of itself and won 100 games to zero.
 

(source)

That AlphaGo Zero had nothing to learn from playing the world's best humans, and that it trounced its artificial parent 100-0, is evolutionary velocity of a majesty not seen since the ectomorphs in the Alien movie franchise. It is also, in its arrogance, terrifying.

DeepMind released 55 games that a previous version of AlphaGo played against itself for Go players around the world to analyze.

Since May, experts have been painstakingly analyzing the 55 machine-versus-machine games. And their descriptions of AlphaGo’s moves often seem to keep circling back to the same several words: Amazing. Strange. Alien.
 
“They’re how I imagine games from far in the future,” Shi Yue, a top Go player from China, has told the press. A Go enthusiast named Jonathan Hop who’s been reviewing the games on YouTube calls the AlphaGo-versus-AlphaGo face-offs “Go from an alternate dimension.” From all accounts, one gets the sense that an alien civilization has dropped a cryptic guidebook in our midst: a manual that’s brilliant—or at least, the parts of it we can understand.
 
[...]
 

Some moves AlphaGo likes to make against its clone are downright incomprehensible, even to the world’s best players. (These tend to happen early on in the games—probably because that phase is already mysterious, being farthest away from any final game outcome.) One opening move in Game One has many players stumped. Says Redmond, “I think a natural reaction (and the reaction I’m mostly seeing) is that they just sort of give up, and sort of throw their hands up in the opening. Because it’s so hard to try to attach a story about what AlphaGo is doing. You have to be ready to deny a lot of the things that we’ve believed and that have worked for us.”

 

Like others, Redmond notes that the games somehow feel “alien.” “There’s some inhuman element in the way AlphaGo plays,” he says, “which makes it very difficult for us to just even sort of get into the game.”
 

Ke Jie, the Chinese Go master who was defeated by AlphaGo earlier this year, said:

Last year, it was still quite humanlike when it played. But this year, it became like a god of Go.”
 

After his defeat, Ke posted what might be the most poetic and bracing quote of 2017 on Weibo (I first saw it in the WSJ):

“I would go as far as to say not a single human has touched the edge of the truth of Go.”
 

***

When Josh Brown died in his Tesla after driving under a semi, it kicked off a months long investigation into who was at fault. Ultimately, the NHTSA absolved Autopilot of blame. The driver was said to have had 7 seconds to see the semi and apply the brakes but was suspected of watching a movie while the car was in Autopilot.

In this instance, it appeared enough evidence could be gathered to make such a determination. In the future, diagnosing why Autopilot or other self-driving algorithms made certain choices will likely only become more and more challenging as the algorithms rise in complexity.

At times, when I have my Tesla in Autopilot mode, the car will do something bizarre and I'll take over. For example, if I drive to work out of San Francisco, I have to exit left and merge onto the 101 using a ramp that arcs to the left almost 90 degrees. There are two lanes on that ramp, but even if I start in the far left lane and am following a car in front of me my car always seems to try to slide over to the right lane.

Why does it do that? My only mental model is the one I know, which is my own method for driving. I look at the road, look for lane markings and other cars, and turn a steering wheel to stay in a safe zone in my lane. But thinking that my car drives using that exact process says more about my limited imagination than anything else because Autopilot doesn't drive the way humans do. This becomes evident when you look at videos showing how a self-driving car "sees" the road.

When I worked at Flipboard, we moved to a home feed that tried to select articles for users based on machine learning. That algorithm continued to be to tweaked and evolved over time, trying to optimize for engagement. Some of that tweaking was done by humans, but a lot of it was done by ML.

At times, people would ask why a certain article had been selected for them? Was it because they had once read a piece on astronomy? Dwelled for a few seconds on a headline about NASA? By that point, the algorithm was so complex it was impossible to really offer an explanation that made intuitive sense to a human, there were so many features and interactions in play.

As more of the world comes to rely on artificial intelligence, and as AI makes great advances, we will walk to the edge of a chasm of comprehension. We've long thought that artificial intelligence might surpass us eventually by thinking like us, but better. But the more likely scenario, as recent developments have shown us, is that the most powerful AI may not think like us at all, and we, with our human brains, may never understand how they think. Like an ant that cannot understand a bit about what the human towering above them is thinking, we will gaze into our AI in blank incomprehension. We will gaze into the void. The limit to our ability to comprehend another intelligence is our ability to describe its workings, and that asymptote is drawn by the limits of our brain, which largely analogizes all forms of intelligence to itself in a form of unwitting intellectual narcissism.

This is part of the general trend of increasing abstraction that marks modern life, but it is different than not knowing how a laptop is made, or how to sew a shirt for oneself. We take solace in knowing that someone out there can. To admit that it's not clear to any human alive how an AI made a particular decision feels less like a ¯\_(ツ)_/¯ and more like the end of some innocence.

I suspect we'll continue to tolerate that level of abstraction when technology functions as we want it to, but we'll bang our heads in frustration when it doesn't. Like the annoyance we feel when we reach the limits of our ability to answer a young child who keeps asking us "Why?" in recursive succession, this frustration will cut deep because it will be indistinguishable from humiliation.

Evaluating mobile map designs

I saw a few links to this recent comparison by Justin O'Beirne of the designs of Apple Maps vs. Google Maps. In it was a link to previous comparisons he made about a year ago. If you're into maps and design, it's a fairly quick read with a lot of useful time series screenshots from both applications to serve as reference points for those who don't open both apps regularly.

However, the entire evaluation seems to come from a perspective at odds with how the apps are actually used. O'Beirne's focus is on evaluating these applications from a cartographic standpoint, almost as if they're successors to old wall-hanging maps or giant road atlases like the ones my dad used to plot out our family road trips when we weren't wealthy enough to fly around the U.S. 

The entire analysis is of how the maps look when the user hasn't entered any destination to navigate to (what I'll just refer to as the default map mode). Since most people use these apps as real-time navigation aids, especially while driving, the views O'Beirne dissects feel like edge cases (that's my hypothesis, of course; if someone out there who has actual data on % of time these apps are used for navigation versus not, I'd love to hear it, even if it's just directional to help frame the magnitude).

For example, much of O'Beirne's ink is spent on each application's road labels, often at really zoomed out levels of the map. I can't remember the last time I looked at any mobile mapping app at the eighth level of zoom, I've probably only spent a few minutes of my life in total in all of these apps at that level of the geographic hierarchy, and only to answer a trivia question or when visiting some region of the world on vacation.

What would be of greater utility to me, and what I've yet to find, is a design comparison of all the major mapping apps as navigation aids, a dissection of the UX in what I'll call their navigation modes. Such an analysis would be even more useful if it included Waze, which doesn't have the market share of Apple or Google Maps but which is popular among a certain set of drivers for its unique approach to evaluating traffic, among other things.

Such a comparison should analyze the visual comprehensibility of each app in navigation mode, which is very different from their default map views. How are roads depicted, what landmarks are shown, how clear is the selected path when seen only in the occasional sidelong glance while driving, which is about as much visual engagement as a user can offer if operating a 3,500 pound vehicle. How does the app balance textual information with the visualization of the roads ahead, and what other POI's or real world objects are shown? Waze, for example, shows me other Waze users in different forms depending on how many miles they've driven in the app and which visual avatars they've chosen.

Of course, the quality of the actual route would be paramount. It's difficult for a single driver to do A/B comparisons, but I still hope that someday someone will start running regular tests in which different cars, equipped with multiple phones, each logged into different apps, try to navigate to the same destination simultaneously. Over time, at some level of scale, such comparison data would be more instructive than the small sample size of the occasional self-reported anecdote.

[In the future, when we have large fleets of self-driving cars, they may produce insights that only large sample sizes can validate, like UPS's "our drivers save time by never turning left." I'd love if Google Maps, Apple Maps, or Waze published some of what they've learned about driving given their massive data sets, a la OKCupid, but most of what they've published publicly leans towards marketing drivel.]

Any analysis of navigation apps should also consider the voice prompts: how often does the map speak to you, how far in advance of the next turn are you notified, how clear are the instructions? What's the signal to noise? What are the default wording choices? Syntax? What voice options are offered? Both male and female voices? What accents?

Ultimately, what matters is getting to your destination in the safest, most efficient manner, but understanding how the applications' interfaces, underlying data, and algorithms influence them would be of value to so many people who now rely on these apps every single day to get from point A to B. I'm looking for a Wirecutter-like battle of the navigation apps, may the best system win.

    The other explicit choice O'Beirne makes is noted in a footnote:

    We’re only looking at the default maps. (No personalization.)
     

    It is, of course, difficult to evaluate personalization of a mapping app since you can generally only see how each map is personalized for yourself. However, much of the value of Google Maps lies in its personalization, or what I suspect is personalization. Given where we are in the evolution of many products and services, analyzing them in their non-personalized states is to disregard their chief modality.

    When I use Google Maps in Manhattan, for example, I notice that that the only points of interest (POI's) the map shows me at various levels of zoom seem to be places I've searched for most frequently (this is in the logged in state, which is how I always use the app). Given Google's reputation for being a world leader in crunching large data sets, it would be surprising if they weren't selecting POI labels, even for non-personalized versions of their maps, based on what people tend to search for most frequently.

    In the old days, if you were making a map to be hung on the wall, or for a paper map or road atlas, what you chose as POI's would be fixed until the next edition of that map. You'd probably choose what felt like the most significant POI's based on reputation, ones that likely wouldn't be gone before the next update. Eiffel Tower? Sure. Some local coffee shop? Might be a Starbucks in three months, best leave that label off.

    Now, maps can be updated dynamically. There will always be those who find any level of personalization creepy, and some are, but I also find the lack of personalization to be immensely frustrating in some services. That I search for reservations in SF on Open Table and receive several hundred hits every time, sorted in who knows what order, instead of results that cluster my favorite or most frequently booked restaurants at the top, drives me batty.

    When driving, personalization is even more valuable because it's often inconvenient or impossible to type or interact with the device for safety reasons. It's a great time saver to have Waze guess where I'm headed automatically ("Are you driving to work?" it asks me every weekday morning), and someday I just want to be able to say "give me directions to my sister's" and have it know where I'm headed.

    My quick first person assessment, despite the small sample size caveats noted earlier:

    • I know that Apple Maps, as the default on iOS, has the market share lead on iPhone by a healthy margin. Still, I'll never get past the time the app took me off to a dead end while I was on the way to a wedding, and I've not used it since except to glance at the design. It may have the most visually pleasing navigation mode aesthetic, but I don't trust their directions at the tails. Some products are judged not on their mean outcome but their handling of the tails. For me, navigation is one of those.
    • It's not clear if Apple Maps should have a data edge over Google Maps and Waze (Google bought Waze but has kept the app separate). Most drivers use it on the iPhone because it's the default, but Google got a headstart in this space and also has a fleet of vehicles on the road taking Google street photos. Eventually, Google may augment that fleet with self-driving cars.
    • I trust Google Maps directions more than those of Apple Maps. However, I miss the usability of the first version of Google Maps, which came out on iOS way back with the first iPhone. I'd heard rumors Apple built that app for Google, but I'm not sure if that's true. The current flat design of Google Maps often strands me in a state in which I have no idea how to initiate navigation. I'd like to believe I'm a fairly sophisticated user and yet I sometimes sit there swiping and tapping in Google Maps like an idiot, trying to get it to start reading turn by turn directions. Drives me batty.

    I use Waze the most when driving in the Bay Area or wherever I trust that there are enough other drivers using Waze that it will offer the quickest route to my destination. That seems true in most major metropolitans. I can tell a lot of users in San Francisco use Waze because sometimes, when I have to drive home to the city from the Peninsula, I find myself in a line of cars exiting the highway and navigating through some random neighborhood side street, one that no one would visit unless guided by an algorithmic deity. 

    I use Waze with my phone mounted to one of those phone clamps that holds the phone at eye level above my dashboard because the default Tesla navigation map is still on Google Maps and is notoriously oblivious to traffic when selecting a route and estimating an arrival time. Since I use Waze more than any other navigation app, I have more specific critiques.

    • One reason I use Waze is that it seems the quickest to respond to temporary buildups of traffic. I suspect it's because the UI has a dedicated, always visible button for reporting such traffic. Since I'm almost always the driver, I have no idea how people are able to do such reporting, but either a lot of passengers are doing the work or lots of drivers able to do so while their car is stuck in gridlock. The other alternative, that drivers are filing such reports while their cars are in motion, is frightening.
    • I don't understand the other social networking aspects of Waze. They're an utter distraction. I'm not immune to the intrinsic rewards of gamification, but in the driving context, where I can't really do much more than glance at my phone, it's all just noise. I don't feel a connection to the other random Waze drivers I see from time to time in the app, all of which are depicted as various pastel-hued cartoon sperm. In wider views of the map, all the various car avatars just add a lot of visual noise.
    • I wish I could turn off some of the extraneous voice alerts, like "Car stopped on the side of the road ahead." I'm almost always listening to a podcast in the background when driving, and the constant interruptions annoy me. There's nothing I can do about a car on the side of the road, I wish I could customize which alerts I had to hear.
    • The ads that drop down and cover almost half the screen are not just annoying but dangerous as I have to glance over and then swipe them off the screen. That, in and of itself, is disqualifying. But beyond that, even while respecting the need for companies to make money, I can't imagine these ads generate a lot of revenue. I've never looked at one. If the ads are annoying, the occasional survey asking me which ads/brands I've seen on Waze are doubly so. With Google's deep pockets behind Waze, there must be a way to limit ads to those moments where they're safe or clearly requested, for example when a user is researching where to get gas or a bit to eat. When a driver has hands on the wheel and is guiding a giant mass of metal at high velocity, no cognitive resources should be diverted to remembering what brands you recall seeing on the app.
    • Waze still doesn't understand how to penalize unprotected left turns, which are almost completely unusable in Los Angeles at any volume of traffic. At rush hour it's a fatal failure, like being ambushed by a video game foe that can kill you with one shot with no advance warning. As long as it remains unfixed, I use Google Maps when in LA. I can understand why knowledge sharing between the two companies may be limited by geographic separation despite being part of the same umbrella company, but that the apps don't borrow more basic lessons from other seems a shame.
    • I use Bluetooth to listen to podcasts on Overcast when driving, and since I downloaded iOS 11, that connection has been very flaky. Also, if I don't have the podcast on and Waze gives me an voice cue, the podcast starts playing. I've tried quitting Overcast, and the podcast still starts playing every time Waze speaks to me. I had reached a good place in that Overcast would pause while Waze spoke so they wouldn't overlap, but since iOS 11 even that works inconsistently. This is just one of the bugs that iOS 11 has unleashed upon my phone, I really regret upgrading.