What's wrong with The Lean Startup?

In the middle of Google Code Jam qualification round, I got sidetracked into pondering startup and entrepreneurship issues. It all started with one local blogger recommending Eric Ries's book The Lean Startup.

I've got so immersed in the book that I lost a lot of ranking in Google Code Jam due to time penalties. I've solved all the problems nevertheless and proceeded to the next round. So back to the book. I've always wanted to learn the big secrets of the science of startups. While this book provides interesting insights that line surprisingly well with what I know about proper research, it should be taken with a grain of salt as I will soon explain.

The first thing that irked me is that someone is instructing me on scientific methods and rigorous empirical evaluation while supporting his own arguments with anecdotal evidence and so-called "plausible" arguments. Why didn't Ries try to evaluate effectiveness of his own method? Just take two groups of startups, couch one of the groups on lean startup methods, then compare success rate with the control group. With Ries doing seminars and consulting around the globe, this couldn't be any easier.

The main argument of the book is that startups are all about chasing metrics. Ries goes into some details on what's a proper metric and how important it is to increase throughput of the research/evaluation pipeline at the cost of preplanned production efficiency.

Essentially, in his view, startups perform long sequences of micro-optimizations. Since micro-optimizations usually converge towards some ceiling that prevents them from moving metrics any further, startups regularly "pivot", i.e. introduce macro-optimizations that raise the ceiling of micro-optimizations at the cost of temporarily slashing metrics down.

With this point of view, it becomes apparent that Ries is introducing inefficiency by delaying macro-optimizations into infrequent pivots. It's much better idea to try tons of macro-optimization early in the project and perform all the micro-optimizations afterwards. This is possible, because (1) macro-optimizations can be often additively combined with each other and (2) effect of macro-optimizations is visible even without delving into the lengthy micro-optimization process.

So if you are micro-optimizing your conversion funnel right now, stop and think about it. Isn't there a better use for your time? Don't you have some radical idea sitting in the backlog just because you are afraid that it could kill all the micro-optimizations you have already done?

The most real and serious crime that Ries commits is talking down classical planning. He essentially dismisses the whole concept of planning as irrelevant for startups. Fail. Big fail. This is gonna kill lots of companies. I'll shortly explain why.

There's this concept of progressive evaluation. Full-size empirical tests are expensive. You want to make sure that only the highest quality hypotheses reach the empirical test pipeline. You run tons of hypotheses through cheaper tests, then run survivors through the expensive tests.

What are these cheaper tests? Imaginary thought experiments, external knowledge (essentially learning from mistakes of others), internal metrics & tests, various automated metrics that do not irreversibly consume test subjects. And of course planning. The point of planning is to optimize the plan. "Just do it" approach is quite inefficient as Ries would agree. Plan allows you to see stupid ideas and to weed out mistakes and inefficiencies before even attempting to build anything.

Yeah, of course, I know you need predictable environment to build a plan. But Ries is taking it to extreme when he claims that any plan in a startup is pure imagination. The trick here is that the world of startups is partially predictable. The plans are not accurate, but they still approximate future events. People are quite skilled at making predictions in complex, partially observable environments. Such imperfect plans are still quite effective at killing tons of stupid mistakes.

Anyway, the main value of the book is that it got me thinking. I can understand Ries's arguments and they are often enlightening, but it's still a very small part of the puzzle. I look with suspicion at the startup world as I generally dislike what money does to technology, but many ideas are applicable in opensource and non-profits too. IMO, rather than wasting your time on website micro-optimizations, you better think how to create something useful and valuable. Yeah, the input from customers will impact your project a lot, but you are the expert and you are responsible for interpreting not just the customer feedback, but also tons of other information at your disposal.