Innovation in organization design

For example, Uber has a mobile app (UI) that talks to their servers (API). You can imagine that their servers effectively take three parameters: credit card, drive from, and drive to… and they dispatch a human to do it.
 

uber.drive(card, pointA, pointB); // pseudocode obviously
 

What does that make the drivers? Cogs in a giant automated dispatching machine, controlled through clever programming optimizations like surge pricing? Drivers have often told me that the job grants them incredible autonomy: they can drive whenever they feel like it, and they’ve stopped looking for jobs in finance or construction because the daily freedom is so valuable to them. There’s liquidity in the marketplace that allows them to come and go as they see fit. But the actual driving is perfectly orchestrated by software, and it’s not a secret that Uber intends to eventually replace all their drivers with self-driving cars. I worry that the army of Lyft and Uber drivers is opting into an easy, and sometimes-intended-to-be-temporary, dead-end career path. This may be ok at the moment for some drivers who enjoy driving and the flexibility of the job. But driving as an occupation will disappear practically overnight when self-driving cars hit the road.
 

Similarly, 99designs Tasks has a web interface for the customer to explain a simple and quick design task, plus an API to dispatch a visual designer to complete the task. At Segment we’ve actually built a 99designs Tasks API to create vector logos from an image url:
 

99designs.logo(card, url); // pseudocode ;)
 

What’s bizarre here is that these lines of code directly control real humans. The Uber API dispatches a human to drive from point A to point B. And the 99designs Tasks API dispatches a human to convert an image into a vector logo (black, white and color). Humans are on the verge of becoming literal cogs in a machine, completely anonymized behind an API. And the companies that control those APIs have strong incentives to drive down the cost of executing those API methods.
 

Peter Reinhardt wrote this a while back on replacing middle managers with APIs. It speaks to a broader trend, but depicting this gaping skills gap as an API call is an elegant visualization.

RankBrain

For the past few months, a “very large fraction” of the millions of queries a second that people type into the company’s search engine have been interpreted by an artificial intelligence system, nicknamed RankBrain, said Greg Corrado, a senior research scientist with the company, outlining for the first time the emerging role of AI in search.
 

RankBrain uses artificial intelligence to embed vast amounts of written language into mathematical entities -- called vectors -- that the computer can understand. If RankBrain sees a word or phrase it isn’t familiar with, the machine can make a guess as to what words or phrases might have a similar meaning and filter the result accordingly, making it more effective at handling never-before-seen search queries.
 

[...]
 

RankBrain is one of the “hundreds” of signals that go into an algorithm that determines what results appear on a Google search page and where they are ranked, Corrado said. In the few months it has been deployed, RankBrain has become the third-most important signal contributing to the result of a search query, he said.
 

[...]
 

So far, RankBrain is living up to its AI hype. Google search engineers, who spend their days crafting the algorithms that underpin the search software, were asked to eyeball some pages and guess which they thought Google’s search engine technology would rank on top. While the humans guessed correctly 70 percent of the time, RankBrain had an 80 percent success rate.
 

More on RankBrain here.

Machine learning is advancing fast. At its best it feels a bit like magic, and it's endlessly malleable. Think it's missing something of importance? Add it as a factor, or tune it up.

I suspect in my lifetime we'll have machine learning so good it will be largely incomprehensible to me. That is, it won't be understandable by using analogies to how humans think because it will be its own form of intelligence.

Data mining algorithms in plain English

Maybe not interesting if you're a data mining guru, but this explanation of the top 10 most influential data mining algorithms in plain English is a good read for the rest of us, though “plain English” is perhaps debatable.

Here's a good one, on k-means:

You might be wondering:
 

Given this set of vectors, how do we cluster together patients that have similar age, pulse, blood pressure, etc?
 

Want to know the best part?
 

You tell k-means how many clusters you want. K-means takes care of the rest.
 

How does k-means take care of the rest? k-means has lots of variations to optimize for certain types of data.
 

At a high level, they all do something like this:

1. k-means picks points in multi-dimensional space to represent each of the k clusters. These are called centroids.

2. Every patient will be closest to 1 of these k centroids. They hopefully won’t all be closest to the same one, so they’ll form a cluster around their nearest centroid.

3. What we have are k clusters, and each patient is now a member of a cluster.

4. k-means then finds the center for each of the k clusters based on its cluster members (yep, using the patient vectors!).

5. This center becomes the new centroid for the cluster.

6. Since the centroid is in a different place now, patients might now be closer to other centroids. In other words, they may change cluster membership.

7. Steps 2-6 are repeated until the centroids no longer change, and the cluster memberships stabilize. This is called convergence.
    

This seems like a great idea for a book: the central data algorithms of the third industrial revolution, this networked, online age. One chapter per algorithm, with a discussion of how it manifests itself on the key websites, applications, hardware, and other services we use all the time now. If you are a data mining expert in need of someone to be the “plain English” side of a writing team, call me maybe.

Robots take all the jobs (composer edition)

Xhail is a new service that offers a unique, custom score for your movie.

Here's the rub: the score is written by software, using real instrument stems. Instead of talking to a composer about what you want, you simply type in keywords like “fantasy” or “melancholy” and the software returns a score which you can customize using the interface provided. Add instruments, take out sections, add percussive emphasis at key timecode to match action on screen. The demo video gives a good sense of how it works.

Lots of details are still missing, like how much does it cost? Still, it's an impressive demo. The track composed for the fantasy short at the end of the demo video and the interface for modifying the video both were much better than I expected. You'd expect nothing less from a scripted demo video, and we'll have to wait for a public release to see if it's all that, but I'm intrigued.

I suspect many will rush to dismiss this service, especially my friends in the filmmaking world, just as people tend to do with any computer-generated art, but some of that, as always, comes from either a general technophobia or reverence for human creation.

If you can afford a real composer, this isn't a service targeted at you. Facetious title of my post aside, I suspect this is a less a case of replacing our existing composer supply than adding supply at the low end of the market.

The automatic corporation?

The intermediate step to a fully automated corporation is one where tasks requiring humans are performed not by employees but are broken into micro-tasks and fulfilled by crowdsourcing (using, for example, services like Mechanical Turk).

Corporations do not scale, and eventually die. That’s because they scale sub-linearly. Their productivity metrics scale by an exponent of ⅘ on the number of employees.

I hypothesize that the management overhead which makes corporations grow sub-linearly is due to the limited information processing capability of individual humans. People at the top do not have local on-the-ground information: how are individual products performing, what are customers’ complaints etc. And the rank-and-file folks on the ground do not have the relevant high-level information: how does what I’m doing translate to the value that the corporation as a whole seeks to maximize? In fact, the the flow of value and information is so complex that employees have pretty much given up on determining that relationship, and know of it only at a macro P&L-center level.

An algorithm will have no such problems with acting on both global as well as fine-grained local information. In fact, I suspect that the more information it gets to act on, the better decisions it will make, making automatic corporations grow super-linearly.
 

More here, all fascinating, on the concept of an automatic corporation.

When the idea of two-pizza teams was first proposed at Amazon, it was an attempt at accomplishing two thing simultaneously. On the one hand, keeping teams small was an attempt at giving them autonomy in figuring out what strategy and projects to pursue. On the other hand, since each team had to optimize on a fitness function agreed upon with senior management, it was a model for scaling Jeff Bezos and his senior management team's ability to coordinate activities across the company. If you have a limited number of people you trust to choose the fitness functions, that's still a bottleneck.

The idea of an automatic corporation would replace the humans in both the fitness function and project selection process with software which scales infinitely where humans cannot.

This may sound far-fetched, but the author Vivek Haldar notes it already exists in some forms today.

A limited version of what I’m describing already exists. High-frequency trading firms are already pure software, mostly beyond human control or comprehension. The flash crash of 2010 demonstrated this. Companies that are centered around logistics, like FedEx or Walmart, can be already thought of as complex software entities where human worker bees carry out the machine’s instructions.

This happens naturally, because over time more and more of the business logic of a company becomes encoded in software. Humans still have some control (or so they think) but mostly what they’re doing is supplying parameters to the computation. A modern corporation is so complex that it does not fit in the brain of a single person (or a small number of persons). Software carries the slack.