The failure of software development methodologies

I’ve worked on big projects, small projects, in huge teams and by myself, in fossilized federal agencies and cool Silicon Valley companies. I have learned and used at least twenty programming languages. I’ve lived through waterfall/BDUF (big design up front), structured programming, top-down, bottom-up, modular design, components, agile, Scrum, extreme, TDD, OOP, rapid prototyping, RAD, and probably others I’ve forgotten about. I’m not convinced any of these things work.

*****

Whether a methodology works or not depends on the criteria: team productivity, happiness, retention, conformity, predictability, accountability, communication, lines per day, man-months, code quality, artifacts produced, etc. Every methodology works if you measure the right thing. But in terms of the only measurement that really matters—satisfying requirement on time and within budget—I haven’t seen any methodology deliver consistent results.

My own experiences are anecdotal, but they are shared by almost every programmer I know. It turns out that anecdotes are all that anyone has: rigorous studies of software development methodologies haven’t been done because it’s impossible to control for all of the variables.

Try this thought experiment: Imagine two teams of programmers, working with identical requirements, schedules, and budgets, in the same environment, with the same language and development tools. One team uses waterfall/BDUF, the other uses agile techniques. It’s obvious this isn’t a good experiment: The individual skills and personalities of the team members, and how they communicate with each other, will have a much bigger effect than the methodology.
 

Thought-provoking. The author concludes:

I think programmers should pay much more attention to listening to and working with their peers than to rituals and tools, and that we should be skeptical of too much process or methodologies that promise to magically make everyone more productive. Maybe social skills come harder to programmers than to other people (I’m not convinced that’s true), but developing those skills will certainly pay off a lot more than trying yet another development methodology.
 

Maybe software development methodologies are like diets, to use an analogy my coworker Eric brought up. Endlessly appealing, rarely successful. Or perhaps they're like workouts. What's needed is variety, and sometimes you just need a new routine to keep things fresh, regardless of what routine you choose.

The Super Bowl, the outsider's description

The ethics of such an event can be hard for outsiders to understand. Fans, who regularly watch players being carted off the field with crippling injuries, are unbothered by reports of the game's lasting medical impact on its players. Nevertheless, fans and the national media can become extremely indignant if players are excessively boastful at the game’s conclusion.

Perhaps in homage to the country’s patriarchal culture, women are generally involved only as scantily clad dancers during breaks in the action. Minority rights groups have also criticized the owner and fans of one of the country’s most popular teams—the one representing the national capital, in fact—for referring to players using a racial nickname too offensive to be printed in this newspaper. Fans of the team, like those of Tottenham Hotspur, have defended the name, saying it is a term of affection.
 

From Slate, “the latest installment of a continuing series in which American events are described using the tropes and tone normally employed by the American media to describe events in other countries.”

Hackers

Interview with Arthur Chu, famous now for studying Jeopardy carefully and developing a strategy to optimize his chances to win. Some are criticizing his playing style, accusing him of hacking the system, but isn't that how one should try to play every game? Learn the rules and develop the optimal strategy?

I noticed some interesting things. Everyone's talking about my strategy on the show, it seems, but I didn't make anything up—I just read people's observations online. In 1985, the second year of the Alex Trebek version of the show, this guy Chuck Forrest really dominated by bouncing around the categories, and they call it the "Forrest Bounce." There's no logical reason to do what people normally do, which is to take one category at a time from the top down. Your only point of control in the game is your ability, if you get the right answer to a question, to select the next question—and you give that power up if you make yourself predictable. The more unpredictable you are, the more you put your opponents off-balance, the longer you can keep an initial advantage. Multiple people over the years have used [the Forrest Bounce] and yet most people haven't used it. When they programmed the computer Watson to maximize its chances of winning, it did the Forrest Bounce. And it specifically did Daily Double hunting. Watson knew that the bottom two rows of the Jeopardy! board are more likely to contain Daily Doubles, and it knew that whoever gets the most Daily Doubles is the most likely to win the game—that's just statistical analysis. So it was programmed to hunt those Daily Doubles, and I figured I had no reason not to do that.
 

Another excerpt of interest:

There are a few specific composers they want [you] to know. If they mention "a Norwegian composer"—this happened in a game, I think the Wednesday game—it will be Edvard Grieg. That's the method they use to write the clue. If they mention a "Polish Nobel Prize Winner," it's likely to be Marie Curie. If they mention a "Female Nobel Prize Winner," it's very likely to be Marie Curie. Jeopardy! is aimed at the sort of average TV viewer, so they're not going to ask things that are pointlessly obscure, they're not going to go in-depth on any particular subject, they're going to focus on these cultural touchstones that we all know.And if you watch the show, and you can identify those, you can literally make flashcards.

So I used a program called Anki which uses a method called "spaced repetition." It keeps track of where you're doing well or poorly, and pushes you to study the flashcards you don't know as well, until you develop an even knowledge base about a particular subject, and I just made flashcards for those specific things. I memorized all the world capitals, it wasn't that hard once I had the flashcards and was using them every day. I memorized the US State Nicknames (they're on Wikipedia), memorized the basic important facts about the 44 US Presidents. I really focused on those. But there's a lot more stuff to know. I went onJeopardy! knowing that there was stuff I didn't know. For instance, everyone laughs about sports—but I also knew that [sports clues] were the least likely to come up in Double Jeopardy and Final Jeopardy and be very important. So I decided I shouldn't sweat it too much, I should just recognize that I didn't know them and let that go, as long as I can get the high value clues.
 

*****

Gamifying world news instruction. Though gamification has some negative connotations thanks to many mobile games and online dark patterns of UI design, so perhaps we should refer to this as some variant of a nudge?

*****

The guy who hacked OkCupid to find love.

But mathematically, McKinlay’s compatibility with women in Los Angeles was abysmal. OkCupid’s algorithms use only the questions that both potential matches decide to answer, and the match questions McKinlay had chosen—more or less at random—had proven unpopular. When he scrolled through his matches, fewer than 100 women would appear above the 90 percent compatibility mark. And that was in a city containing some 2 million women (approximately 80,000 of them on OkCupid). On a site where compatibility equals visibility, he was practically a ghost.

He realized he’d have to boost that number. If, through statistical sampling, McKinlay could ascertain which questions mattered to the kind of women he liked, he could construct a new profile that honestly answered those questions and ignored the rest. He could match every woman in LA who might be right for him, and none that weren’t.
 

I expected the article to be about how his complex hacking led quickly to the perfect woman, but what's surprising (or depressing, depending on how much faith you put in market efficiency) is how much work he had to go through even after his complex data mining surfaced a candidate list.

Far easier, I suppose, if you are able to nail the three rules mentioned in the old Tom Brady sexual harassment sketch on Saturday Night Live:

1. Be Handsome.
2. Be Attractive.
3. Don't Be Unattractive.

I don't envy the best man at McKinlay's wedding, having to recount the story of how the couple met.

The sofalarity

The Oji-Cree have been in contact with European settlers for centuries, but it was only in the nineteen-sixties, when trucks began making the trip north, that newer technologies like the internal combustion engine and electricity really began to reach the area. The Oji-Cree eagerly embraced these new tools. In our lingo, we might say that they went through a rapid evolution, advancing through hundreds of years of technology in just a few decades.

The good news is that, nowadays, the Oji-Cree no longer face the threat of winter starvation, which regularly killed people in earlier times. They can more easily import and store the food they need, and they enjoy pleasures like sweets and alcohol. Life has become more comfortable. The constant labor of canoeing or snowshoeing has been eliminated by outboard engines and snowmobiles. Television made it north in the nineteen-eighties, and it has proved enormously popular.

But, in the main, the Oji-Cree story is not a happy one. Since the arrival of new technologies, the population has suffered a massive increase in morbid obesity, heart disease, and Type 2 diabetes. Social problems are rampant: idleness, alcoholism, drug addiction, and suicide have reached some of the highest levels on earth. Diabetes, in particular, has become so common (affecting forty per cent of the population) that researchers think that many children, after exposure in the womb, are born with an increased predisposition to the disease. Childhood obesity is widespread, and ten-year-olds sometimes appear middle-aged. Recently, the Chief of a small Oji-Cree community estimated that half of his adult population was addicted to OxyContin or other painkillers.
 

From Tim Wu over at The New Yorker. What is the sofalarity?

Technological evolution has a different motive force. It is self-evolution, and it is therefore driven by what we want as opposed to what is adaptive. In a market economy, it is even more complex: for most of us, our technological identities are determined by what companies decide to sell based on what they believe we, as consumers, will pay for. As a species, we often aren’t much different from the Oji-Cree. Comfort-seeking missiles, we spend the most to minimize pain and maximize pleasure. When it comes to technologies, we mainly want to make things easy. Not to be bored. Oh, and maybe to look a bit younger.

Our will-to-comfort, combined with our technological powers, creates a stark possibility. If we’re not careful, our technological evolution will take us toward not a singularity but a sofalarity. That’s a future defined not by an evolution toward superintelligence but by the absence of discomforts.
 

Ah yes, it's the Wall-E scenario. If we are not careful, technology may lead us to a corrupted Maslow's hierarchy of needs in which the top rung is not self-actualization but the absence of discomfort.

Maybe we will follow a trail of in-app purchases there. Someone who has finished Candy Crush Saga, let me know if, when you complete the game, a video plays showing the developer laughing at you and shouting, “Suckerrrrrrrr!”

Why don't people have more sex?

An older but still amusing post from Marginal Revolution.

3. Freud was right and we are all repressed.  The will is not unitary and the utility-maximizing part is not always in control.

4. There has been a market failure, but the Internet is remedying it.  People are having more sex and this will only go up.

5. Sex stops being fun when you do it to close a gap between your marginal utilities.  It requires spontaneity or some other quality inconsistent with the classical model of the consumer and the equation of marginal rates of substitution.