Learning the Bitter Lesson
Our colleague Jacob shed a light on working with AI in a post he wrote back in June, Contemplations. One of his points was that, with AI being more and more ubiquitous in our work as software developers, sophisticated programming concepts are starting to become irrelevant. The AI doesn’t care how elegant the code is, it spits out something that works. In the end that’s what matters to the business; it just works.
I learned about Rich Sutton’s "Bitter Lesson" recently, and I can’t help but draw parallels between the work on the cutting edge he describes and our daily work using AI as software engineers.
What is the "Bitter Lesson"?
In summary, Sutton describes the Bitter Lesson as our persistence of designing systems by leveraging our human knowledge in a given domain, while the abundance of available computation power ultimately makes generic, computationally intensive solutions more effective. He calls this lesson bitter because it goes against our intuition and our desire to build systems that incorporate our knowledge and understanding of the world.
He identifies two general purpose methods that have proven to be much more effective than any alternative that leverages human knowledge: searching and learning. Our minds are very complex, as they reflect the outside world which is intrinsically complex. Sutton argues it is more effective to focus on using these "meta-methods" that can capture complexity, rather than encode human knowledge in the solution. His bottom line is that we should strive to develop agents that discover in the same way humans can.
How does the Bitter Lesson reflect on our daily work as software engineers?
As software engineers, we are tasked with building, but more importantly, understanding software. I think the Bitter Lesson can give us opportunities to be more effective by decreasing necessity of having to learn existing domain knowledge and having to develop a deep understanding for it, and instead focus more on understanding more abstract and generic methods that leverage computation. What do I mean with that? Let’s use an example. Say you are building a system that needs to process a large amount of data. You could spend a lot of time learning about the specific domain and the best ways to process that data, or you could focus on developing a general method that can process any kind of data efficiently. By focusing on the general method, you can leverage computation to solve the problem more effectively.
As using AI to produce software is already making us many times more productive in terms of LOC, the price of every line has already dropped significantly. I would argue that the bitter lesson is just another argument to face this new reality. Where spending a lot of effort in making a piece of elegant code is just not worth the effort. At least not compared to letting the AI make a piece of generic code that does the job just as effectively, even if it is more computationally intensive.
In conclusion, I think the Bitter Lesson is an argument that we shouldn’t strive to encode human knowledge into a system when our time might be better spent on understanding and building these "meta-methods" that can capture complexity.
