Lean Startup Model: Centering Customer Experience
Lean Startup Model: Centering Customer Experience
I have just finished reading the book The Lean Startup by Eric Ries. It was recommended to me almost a decade ago by food startup founders whom I interviewed for a food startup research project while in grad school. Several times, the book was mentioned as if it was a bible of the startup world. I had delayed a deep-dive into the book until now. Over this holiday season, I made an attempt to pick it up, and read it from cover to cover. The book is about how teams make decision in the face of extreme uncertainties. Some insights from the book are worth examining here in the era of breakneck-speed generative AI developments.
The Vision-Strategy-Product Pyramid
According to Ries, the company’s vision is the foundation that keeps all activities at the startup together. Product, sales, growth strategies are tactical strategies that are built on top of the vision. The concept “pivoting” is an important concept in the startup world. However, Ries doesn’t suggest “pivot” means changing without any direction. It has to be a change conditioned on lessons learned, and based on the broad vision that the company has held. These lessons are discovered using the “lean startup“ methodology which places an emphasis on “validated learning.“ Validated learning is another way to say that for everything that one does, there must be a clear hypothesis, and a way to test that hypothesis, then make a decision based on the results of such a hypothesis testing process. In other words, Ries is advocating for scientific management, and product development. Finally, once a product/feature is created, the rest of the work is in the area of optimization of a product.
The Build-Measure-Learn Feedback Loop
The build-measure-learn feedback loop is simply a product development cycle. A team often goes through product ideation, builds something quickly, measures the effects of the products on the target population, collects necessarily data, and learns from the data analysis. Then the cycle repeats. In this process, the main emphasis is to reduce the amount of time this cycle completes. The earlier an idea can be turned into a product, the faster the team can learn from the target consumers whether there’s a market fit.
The goal to minimize the amount of time for such a build-measure-learn feedback loop to complete is very beneficial in many areas not just for a product development. One might think of it is in a fractal framework, where within each circle like ideas, build, product, measure, data, learn, there’s a mini-build-measure-learn feedback loop. For example, as a data scientist whose products are internal analytical reports, I work with internal stakeholders. When I build something quickly, I can ship my “product,“ which is often an analysis. Then I can measure quickly whether my “consumers,” ie. other departments appreciate, agree with the report. Finally, after getting the feedback from other departments, I can collect the data of their feedback in the form of quantitative analysis or verbal feedback, and then learn from the data. When the cycle is completed, I can get back to refine the idea, or to discard the idea altogether, and move on to the next project.
Critique of the agile framework
One important insight from the book is the concept of “waste.“ According to Ries, any effort that goes into building, ideating which doesn’t directly go into shipping a product early is a waste. What Ries means by waste here is that most people want to ship the most polish products to customers. However, in order to polish a product, a team must make a lot of decisions such as engineering, design decisions that might go into features, aspects of products that customers don’t like. So instead, they should not aim for perfection, but rather completion. Customers should have the “first crappy products“ in hand to be able to provide feedback, critique, and suggestions, whose ideas are more grounded, which the design, engineering team can then act upon. This idea resonates with me as a data scientist, and as a writer.
During my graduate school years, the idea I love the most was to “write the first crappy draft“ for any paper that I was trying to produce: from a simple blog post, to a book chapter, a research article, and then finally the dissertation. At the beginning I was so afraid of producing bad products, I never produced anything at all. Until when I took a class “writing for publication,“ where I learned that most good writers have written bad sentences, and the first draft is the most important draft. Once it is out, there’s something to work on. One can start edit it. They can send it around for feedback. They can even print it out to the tear it apart, burn it, or whatever. But there must be something there, concretely for criticisms. Otherwise, everything is still in one’s head. If it’s not on paper, it’s not an idea. So I got over my fear of producing bad writing. I just wrote one sentence after another. That’s how most writings got done. Then I started sending my writings to others. Most feedback I got at first was not that exciting, but people reacted to my writing. At least I have an audience, albeit an uninterested one. Gradually I felt comfortable writing at longer length. In other words, I agree with Ries’ approach of not wasting anytime in debating, negotiating, attending endless meetings that don’t lead to putting a product or another version of the product into the customer’s hands.
One pitfall that engineering teams often fall into is being stuck in the Build phase. This is where the agile development framework comes in. It is a very popular framework in software development. It privileges the build process. It’s created for engineers to optimize for their productivity without taking into consideration what the customers actually want at the end. Ries argues that the agile framework with story being accepted constantly to move production along is too rigid for a process called validated learning, meaning that each story has to have an end goal in mind: does this story contribute to building a feature that increases customers’ engagement, revenues, etc. In other words, procedural thinking (agile development) privileges productivity without thinking through if this productivity is needed to start with. Who is then to call it out when engineers create stories that are at the end considered as wasteful? The answer is no one person, but it’s rather a system of processes. The answer lies in “validated learning.“ That means, each story after completed has to be validated that the story is indeed helping a customer, most of the time is through hypothesis testing, or A/B testing.
The idea that engineering efforts might be wasteful if their productivity is being acknowledge only by managers, but not necessarily by consumers is very powerful. One contemporary example is the current engagement of engineers with customers on social media about Starfield, a video-game produced by Bethesda. The game has received very negative reviews left and right especially from Bethesda’s long term fans. Most criticism is along the line that the game is boring, the world-building is without depth, and characters are not memorable. Fans have gone on to post their feelings about the game on different sites, game reviews, making Youtube videos about their reviews, etc. Interestingly, developers of the game have come out to defend how hard it was to create this game, to build features, to design a game engine, etc.
Of course, game players won’t understand how hard it is to make each piece of the entire game. But what matters is that the consumers, i.e. gamers don’t think highly of the game even after all those efforts that engineers put in. What we are seeing are a few issues at play: (1) the developers were working diligently for 5 years to release a beautiful product that they hope customers will enjoy; (2) customers (mostly Bethesda fans) are expecting some wow experience that will draw them back in; (3) customers’ feedback is so negative, and the cultural phenomenon has become so big that everyone wants to make their side clear of why the game has gone completely opposite of what everyone had hoped for. From Lean Startup philosophy point of view, everything that doesn’t go into how a customer can positively experience the game is a waste. What it means is that a lot of the time engineers tried to figure out how to create a feature was a waste. What puzzled me even further is why it takes so long to build a game? 5 years without having proper mainstream customers feedback baked in during the process seems like a very risky business. Is there a possible way of building a game that takes a lot less time? Or if it has to take that long, can user testing (with mainstream users) be done more often in the building process?
Now what are the lessons for creating products in the Generative AI era? I would argue that following the lean startup philosophy is the way to go for both startups as well as large corporations. Since the field is moving so quickly, it is important to have products go to consumers quickly. Only when the customers can experience what the effects that generative AI can do for them, then a team can decide if their product has any market fit.