We can’t always keep digging in the same hole.

I recently finished reading Edward de Bono’s Lateral Thinking. I won’t bore you by trying to summarize all of de Bono’s points but it’s an interesting book and I would recommend it to anyone that is looking for new ways to solve problems. I will however, highlight one particular point from this book that I see over and over again. de Bono believes that it is impossible to look in a new direction while looking HARDER in the same direction. He describes this via a metaphor of digging a hole:

Let’s say that you are looking for buried treasure, and you start by digging a hole in the place where you think that the treasure is to be found. At some point you are likely to realize that the treasure isn’t there, but it is easier in de Bono’s opinion to keep digging in the same hole than stopping the digging and starting a new hole. I see this problem all the time when it comes to solving complex technical problems with software. It often starts out innocently enough. Developers put together a prototype to show how a particular problem might be solved—and good or bad that prototype ends up morphing into the finished product. Proponents of agile project management practices tell us that all development should consist of a certain amount of ongoing refactoring. But in practice refactoring isn’t always given the focus that it needs to have and refactoring can’t always solve the problem of digging in the wrong place. The three little pigs didn’t refactor their straw house into something stronger, they built a whole new brick house.
We’ve been struggling with this problem recently at one of our companies. The team built out a portion of the new product using a NoSQL document store as the primary database. We’ve had real success using MongoDB, at another of our companies. In this particular case, the concept was a good one and the reasons for using a document store database are sound. While we don’t have availability and scalability problems with this product—aka a “CAP” problem—we do have the object/relational impedance problem in spades here. The issue is that the more that we work with our proposed solution, the more that we are finding that we need to make compromises to solve other parts of our system, particularly reporting. So, we keep digging in the same hole. At some point we are going to have to decide if all of the compromises are worth it. We aren’t there yet, and our current design might just be the “right” approach for this platform, but it’s not clear at this point in the project.

I see this a lot in the mobile maret. Teams start out building HTML-based applications for iOS and Android devices. While this is often the right approach for certain types of applications, sometimes it isn’t. When does the team put down their shovels and think hard about starting a new hole? There is no easy answer, although I do wish that I had a one-size-fits-all fix. The truth is, there isn’t one. The decision to stop digging is a big decision and requires strength of character and conviction that can be awfully hard when you are under the gun to deliver a product. What I have learned is that you can minimize the potential problem with strong communication channels. Technical teams need to be very frank about the state of a solution to a given problem. Trying to “please” the C-suite by trying to morph the prototype into a production release is a bad approach. Most times you need to throw away the prototype. The prototype can help you figure out the ultimate solution, but trying to coerce it into being the production product means that you are simply digging in the same hole. Management needs to understand that there is a certain amount of “research” in “research and development.” You might need to throw away the research in order to make the final product scalable and reliable. What makes this issue so complex is that there are times when the team needs to power through difficult technical issues in order to get to the other side. Yes, there are cases where you are digging in the right place, you just need to dig a little deeper.

As I said, I wish that there was a simple answer to this problem, but there isn’t. What I can tell you is that it’s always good practice to pick your head up out of the hole on a periodic basis and spend a few minutes thinking about whether you are digging in the right place.