When it comes to productivity, there is one set of rules, which economists have worked on since Adam Smith. Innovation has a different set of rules. Most economists seem barely aware that the two sets of rules often clash — what is good for productivity is bad for innovation. Let me sketch a few of the innovation rules. Innovation needs freedom, of course, and the ability to profit from your invention, which I’ll call benefit. It is also called self-interest. The importance of benefit/self-interest for innovation is the main point of Why Nations Fail by Acemoglu and Robinson. Innovation is also increased by resources, such as skills, knowledge, space, and equipment. After discussing this with Bryan Caplan, I believe many economists are well aware these three factors (freedom, benefit, resources) affect innovation. All three also increase productivity — for example, more resources, more productivity. Far fewer economists realize that two other things, which act against productivity, are also very helpful for innovation:
1. Pain. Not a lot — not debilitating or all-consuming pain — but enough to make you think hard. Necessity is the mother of invention is the aphorism, which isn’t quite right. Pain, not necessity. Government is useful here, as I said. So is war. Many innovations came from wars. A famous example is the greenback, which came from the Civil War.
2. Stability. To innovate, you need free time, which is different from freedom (ask any prisoner). Free time allows painless failure, very helpful for innovation. To have free time, you need a secure job. Government is useful here, too. So is tenure. Pain plus stability = peacetime military spending. The internet came from peacetime military spending. Professors were the first users. Stability also promotes innovation because it makes it easier to detect small improvements. The quieter it is, the better you can detect soft sounds.
More here from Seth Roberts. It's commonly accepted that constraints can spur creativity, but the idea of government as a useful irritant is not something I'd heard before.
The tradeoff of freedom and pain with government plays out on a smaller scale with employees and companies. Early in your career, you run into more obstacles in a company given your generally lower position in the organization. Some of them are instructive, others are just friction or the usual coordination costs of an organization.
At some point, for some people, those costs become unbearable and they leave for a position higher up, where there are fewer obstacles, or they start their own company and trade one type of challenge for a different type.