Innovation in organization design

For example, Uber has a mobile app (UI) that talks to their servers (API). You can imagine that their servers effectively take three parameters: credit card, drive from, and drive to… and they dispatch a human to do it.
 
uber.drive(card, pointA, pointB); // pseudocode obviously
 
What does that make the drivers? Cogs in a giant automated dispatching machine, controlled through clever programming optimizations like surge pricing? Drivers have often told me that the job grants them incredible autonomy: they can drive whenever they feel like it, and they’ve stopped looking for jobs in finance or construction because the daily freedom is so valuable to them. There’s liquidity in the marketplace that allows them to come and go as they see fit. But the actual driving is perfectly orchestrated by software, and it’s not a secret that Uber intends to eventually replace all their drivers with self-driving cars. I worry that the army of Lyft and Uber drivers is opting into an easy, and sometimes-intended-to-be-temporary, dead-end career path. This may be ok at the moment for some drivers who enjoy driving and the flexibility of the job. But driving as an occupation will disappear practically overnight when self-driving cars hit the road.
 
Similarly, 99designs Tasks has a web interface for the customer to explain a simple and quick design task, plus an API to dispatch a visual designer to complete the task. At Segment we’ve actually built a 99designs Tasks API to create vector logos from an image url:
 
99designs.logo(card, url); // pseudocode ;)
 
What’s bizarre here is that these lines of code directly control real humans. The Uber API dispatches a human to drive from point A to point B. And the 99designs Tasks API dispatches a human to convert an image into a vector logo (black, white and color). Humans are on the verge of becoming literal cogs in a machine, completely anonymized behind an API. And the companies that control those APIs have strong incentives to drive down the cost of executing those API methods.
 

Peter Reinhardt wrote this a while back on replacing middle managers with APIs. It speaks to a broader trend, but depicting this gaping skills gap as an API call is an elegant visualization.

 

RankBrain

For the past few months, a “very large fraction” of the millions of queries a second that people type into the company’s search engine have been interpreted by an artificial intelligence system, nicknamed RankBrain, said Greg Corrado, a senior research scientist with the company, outlining for the first time the emerging role of AI in search.
 
RankBrain uses artificial intelligence to embed vast amounts of written language into mathematical entities -- called vectors -- that the computer can understand. If RankBrain sees a word or phrase it isn’t familiar with, the machine can make a guess as to what words or phrases might have a similar meaning and filter the result accordingly, making it more effective at handling never-before-seen search queries.
 
[...]
 
RankBrain is one of the “hundreds” of signals that go into an algorithm that determines what results appear on a Google search page and where they are ranked, Corrado said. In the few months it has been deployed, RankBrain has become the third-most important signal contributing to the result of a search query, he said.
 
[...]
 
So far, RankBrain is living up to its AI hype. Google search engineers, who spend their days crafting the algorithms that underpin the search software, were asked to eyeball some pages and guess which they thought Google’s search engine technology would rank on top. While the humans guessed correctly 70 percent of the time, RankBrain had an 80 percent success rate.
 

More on RankBrain here.

Machine learning is advancing fast. At its best it feels a bit like magic, and it's endlessly malleable. Think it's missing something of importance? Add it as a factor, or tune it up.

I suspect in my lifetime we'll have machine learning so good it will be largely incomprehensible to me. That is, it won't be understandable by using analogies to how humans think because it will be its own form of intelligence.

Data mining algorithms in plain English

Maybe not interesting if you're a data mining guru, but this explanation of the top 10 most influential data mining algorithms in plain English is a good read for the rest of us, though “plain English” is perhaps debatable.

Here's a good one, on k-means:

You might be wondering:
 
Given this set of vectors, how do we cluster together patients that have similar age, pulse, blood pressure, etc?
 
Want to know the best part?
 
You tell k-means how many clusters you want. K-means takes care of the rest.
 
How does k-means take care of the rest? k-means has lots of variations to optimize for certain types of data.
 
At a high level, they all do something like this:
  1. k-means picks points in multi-dimensional space to represent each of the k clusters. These are called centroids.
  2. Every patient will be closest to 1 of these k centroids. They hopefully won’t all be closest to the same one, so they’ll form a cluster around their nearest centroid.
  3. What we have are k clusters, and each patient is now a member of a cluster.
  4. k-means then finds the center for each of the k clusters based on its cluster members (yep, using the patient vectors!).
  5. This center becomes the new centroid for the cluster.
  6. Since the centroid is in a different place now, patients might now be closer to other centroids. In other words, they may change cluster membership.
  7. Steps 2-6 are repeated until the centroids no longer change, and the cluster memberships stabilize. This is called convergence.
     

This seems like a great idea for a book: the central data algorithms of the third industrial revolution, this networked, online age. One chapter per algorithm, with a discussion of how it manifests itself on the key websites, applications, hardware, and other services we use all the time now. If you are a data mining expert in need of someone to be the “plain English” side of a writing team, call me maybe.

Robots take all the jobs (composer edition)

Xhail is a new service that offers a unique, custom score for your movie.

Here's the rub: the score is written by software, using real instrument stems. Instead of talking to a composer about what you want, you simply type in keywords like “fantasy” or “melancholy” and the software returns a score which you can customize using the interface provided. Add instruments, take out sections, add percussive emphasis at key timecode to match action on screen. The demo video gives a good sense of how it works.

Lots of details are still missing, like how much does it cost? Still, it's an impressive demo. The track composed for the fantasy short at the end of the demo video and the interface for modifying the video both were much better than I expected. You'd expect nothing less from a scripted demo video, and we'll have to wait for a public release to see if it's all that, but I'm intrigued.

I suspect many will rush to dismiss this service, especially my friends in the filmmaking world, just as people tend to do with any computer-generated art, but some of that, as always, comes from either a general technophobia or reverence for human creation.

If you can afford a real composer, this isn't a service targeted at you. Facetious title of my post aside, I suspect this is a less a case of replacing our existing composer supply than adding supply at the low end of the market.

The automatic corporation?

The intermediate step to a fully automated corporation is one where tasks requiring humans are performed not by employees but are broken into micro-tasks and fulfilled by crowdsourcing (using, for example, services like Mechanical Turk).

Corporations do not scale, and eventually die. That’s because they scale sub-linearly. Their productivity metrics scale by an exponent of ⅘ on the number of employees.

I hypothesize that the management overhead which makes corporations grow sub-linearly is due to the limited information processing capability of individual humans. People at the top do not have local on-the-ground information: how are individual products performing, what are customers’ complaints etc. And the rank-and-file folks on the ground do not have the relevant high-level information: how does what I’m doing translate to the value that the corporation as a whole seeks to maximize? In fact, the the flow of value and information is so complex that employees have pretty much given up on determining that relationship, and know of it only at a macro P&L-center level.

An algorithm will have no such problems with acting on both global as well as fine-grained local information. In fact, I suspect that the more information it gets to act on, the better decisions it will make, making automatic corporations grow super-linearly.
 

More here, all fascinating, on the concept of an automatic corporation.

When the idea of two-pizza teams was first proposed at Amazon, it was an attempt at accomplishing two thing simultaneously. On the one hand, keeping teams small was an attempt at giving them autonomy in figuring out what strategy and projects to pursue. On the other hand, since each team had to optimize on a fitness function agreed upon with senior management, it was a model for scaling Jeff Bezos and his senior management team's ability to coordinate activities across the company. If you have a limited number of people you trust to choose the fitness functions, that's still a bottleneck.

The idea of an automatic corporation would replace the humans in both the fitness function and project selection process with software which scales infinitely where humans cannot.

This may sound far-fetched, but the author Vivek Haldar notes it already exists in some forms today.

A limited version of what I’m describing already exists. High-frequency trading firms are already pure software, mostly beyond human control or comprehension. The flash crash of 2010 demonstrated this. Companies that are centered around logistics, like FedEx or Walmart, can be already thought of as complex software entities where human worker bees carry out the machine’s instructions.

This happens naturally, because over time more and more of the business logic of a company becomes encoded in software. Humans still have some control (or so they think) but mostly what they’re doing is supplying parameters to the computation. A modern corporation is so complex that it does not fit in the brain of a single person (or a small number of persons). Software carries the slack.

How to become a speed reader, updated

Spritzing presents reading content with the ORP located at the specific place where you’re already looking, allowing you to read without having to move your eyes. With this approach, reading becomes more efficient because Spritzing increases the time your brain spends processing content without having to waste time searching for the next word’s ORP. Spritzing also enhances reading on small screens. Because the human eye can focus on about 13 characters at a time, Spritzing requires only 13 characters’ worth of space inside our redicle. No other reading method is designed to help you read all of your content when you’re away from a large screen. But don’t take our word. The following video compares traditional reading to Spritz and is a real eye-opener when it comes to the efficiencies that are gained by placing words exactly where your brain wants them to be located.
 

More here from Spritz Inc. on their speed reading technology. It's worth looking at a demo of the Spritz speed reading aid in action in this article. By placing each word of the text you're reading in a position so that the key letter of each word is located at the same point, your eye doesn't have to move across words on a page. It turns out that eye movement in traditional reading is inefficient. Allowing your eye to stay fixated in one spot increases your reading throughput (though it sounds lazy; don't make my eye have to move even a few millimeters, it's so taxing!).

I took a speed reading course when I was in 6th grade, I was taught that the key to speed reading was to consume blocks of words at a time and to stop yourself from subvocalizing (that is, sounding out the words silently in your head as you read). You can try a number of tricks to cure yourself of that habit, one is to hum to yourself while reading. That blocks your ability to subvocalize.

Spritz's approach to speed reading is a bit different. Rather than scanning groups of words at a time, you're reading one word at a time. I can't imagine reading that way, but everything new seems odd, and every time I find myself rejecting the new I feel like Grandpa Simpson so I'm curious to try this out.

UPDATED: Professor John Henderson is skeptical of Spritz's claims.

So Spritz sounds great, and even somewhat scientific. But can you really read a novel in 90 minutes with full comprehension? Well, like most things that seem too good to be true, the answer unfortunately is no. The research in the 1970s showed convincingly that although people can read using RSVP at normal reading rates, comprehension and memory for text falls as RSVP speeds increase, and the problem gets worse for paragraphs compared to single sentences. One of the biggest problems is that there just isn’t enough time to put the meaning together and store it in memory (what psychologists call “consolidation”). The purported breakthrough use of the “ORP” doesn’t really help with this, and isn’t even novel. In the typical RSVP method, words are presented centered at fixation. The “slightly left of fixation” ORP used by Spritz is a minor tweak at best.

Two other points are worth noting. One is that reading at fast RSVP rates is tiring. It requires unwavering attention and vigilance. You can’t let your mind wander, ponder the nuances of what you’re reading, make a mental note to check on a related idea, or do any other mental activity that would normally be associated with reading for comprehension. If you try, you’ll miss some of the text that is relentlessly flying at you. The second point is that the difficulty of comprehension during reading changes over the course of a sentence, paragraph, and page. Our eyes engage in a choreographed dance through text that reflects this variation in the service of comprehension. RSVP makes every step in the dance the same. Or, to stretch an analogy, imagine hiking along a forest trail. Each step you take determines your overall hiking speed. Some steps require a longer pause to gain footing on loose stones, and others require a longer stride to step over a protruding root. Would it be effective to run on the trail? Worse, would it be a good idea to tie a piece of rope between your ankles so that each step was constrained to be exactly the same length? Surely this would lead to some stumbling, if not to a twisted ankle or catastrophic fall!

The failure of software development methodologies

I’ve worked on big projects, small projects, in huge teams and by myself, in fossilized federal agencies and cool Silicon Valley companies. I have learned and used at least twenty programming languages. I’ve lived through waterfall/BDUF (big design up front), structured programming, top-down, bottom-up, modular design, components, agile, Scrum, extreme, TDD, OOP, rapid prototyping, RAD, and probably others I’ve forgotten about. I’m not convinced any of these things work.

*****

Whether a methodology works or not depends on the criteria: team productivity, happiness, retention, conformity, predictability, accountability, communication, lines per day, man-months, code quality, artifacts produced, etc. Every methodology works if you measure the right thing. But in terms of the only measurement that really matters—satisfying requirement on time and within budget—I haven’t seen any methodology deliver consistent results.

My own experiences are anecdotal, but they are shared by almost every programmer I know. It turns out that anecdotes are all that anyone has: rigorous studies of software development methodologies haven’t been done because it’s impossible to control for all of the variables.

Try this thought experiment: Imagine two teams of programmers, working with identical requirements, schedules, and budgets, in the same environment, with the same language and development tools. One team uses waterfall/BDUF, the other uses agile techniques. It’s obvious this isn’t a good experiment: The individual skills and personalities of the team members, and how they communicate with each other, will have a much bigger effect than the methodology.
 

Thought-provoking. The author concludes:

I think programmers should pay much more attention to listening to and working with their peers than to rituals and tools, and that we should be skeptical of too much process or methodologies that promise to magically make everyone more productive. Maybe social skills come harder to programmers than to other people (I’m not convinced that’s true), but developing those skills will certainly pay off a lot more than trying yet another development methodology.
 

Maybe software development methodologies are like diets, to use an analogy my coworker Eric brought up. Endlessly appealing, rarely successful. Or perhaps they're like workouts. What's needed is variety, and sometimes you just need a new routine to keep things fresh, regardless of what routine you choose.