Did Scrum Kill Agile

written by Geoffrey Greene on 3/27/2023

I remember being shocked at hearing derogatory comments about Scrum from the CTO of a company (let’s call him Bob) during due diligence (the company I was with at the time was acquiring the CTO’s company). Bob and I were CTO to CTO small-talking about development process (we were a Scrum shop) and it was clear that Bob was not a fan. I was surprised at the strength of his conviction that Scrum was trash. Tech people are known for extreme opinions about technology and just about everything else but his denunciation had an edge to it. Up to that point in my career it had been a force for good and had helped me and the teams I lead thrive in the chaos of startup software development. The most colorful complaint I heard from Bob was that Scrum was “corporate bullshit” and for “big, slow, stupid companies”. Obviously this stung since, well, in his view I must be pushing “corporate bullshit” and was happy being “big, slow and stupid” (tbf I needed to lose a few pounds). It took a lot of self-control to not feel this as an attack. My job, after all, was keeping the new team engaged. I ignored my first instinct, which was to get into a philosophical argument with Bob. Acquisitions can be very emotionally difficult for all parties involved. I feared starting our relationship by pushing an agenda with the new team would alienate them and slow everything down. And, in any event, I had no intention of imposing a process on Bob or his team. There wasn’t really an upside to getting too in the weeds on the topic at that moment.

So I moved on but it made me curious about Bob’s reaction. I wondered how wide-spread this scrum-hate was. Narrator voice: it was VERY widespread.

First though, why, at that time, was I such a fanboy of Scrum? I became a practitioner during my time as a CTO at a well known Boston-based startup, in the mid/late 2000s. Scrum wasn’t yet mainstream but it was gaining considerable traction. Both Scrum and the Agile Manifesto were introduced around 2001-2002. Before we adopted Scrum we were using POW (Plain Old Waterfall). It was how development teams operated in the 90’s and early 2000’s when I was an engineer working in commercial software shipping on disks. I simply used the tools I knew for SaaS, which meant a team made up of product managers, QA, build teams and IT/OPS folks. We had lots of handoffs and gating. We were very busy and going in a million different directions. It was a typical growing internet startup, full of smart technical folks but being held back by an antiquated process. Seemingly out of nowhere, someone on the team suggested we do a Scrum training offsite. I didn’t realize how much pain the team was feeling until that moment when I finally confronted the fact that the suggestion to do the training wasn’t out of nowhere. POW had outlived its utility and the team was telling me we had to work in a different way. The training was eye-opening. Scrum seemed to be the answer to all of the struggles we were experiencing.

Before Scrum we did all the standard POW stuff:

Upfront planning and analysis - decreased probability of delivering valuable software
Big releases - high risk, unknown quality
Upfront Estimation and deadlines - stress and effort for engineering for little business value
Crunchtime - not sustainable, bad for people, bad for the culture
Hero engineering - incentivizing firefighters created lots of fires
Handoffs between teams - pure waste
Infrequent and shallow customer involvement - easy to deliver the wrong product

Predictably, a lot of the software we produced under these conditions wasn’t very good. It took too long, had too many defects and was often not really solving customer problems. And the developer experience sucked. In my defense, SaaS was still a new thing and many teams across the industry were struggling with the different cadence of writing software in an always-on world. Shipping software in a box every 12 to 18 months does not prepare you for the WWW. Many of the concepts from Scrum are now boringly mainstream - it’s difficult to convey how radical - at least to me - they seemed in the early 2000s.

Post POW aspirations:

MVPs - solve the problem with the least effort
Small releases - emphasize simplicity, build quality in
Continuous delivery - get value to/feedback from customers ASAP
Sustainable pace - no crunchtime, no burnout
Teamwork - work together and self-organize
Cross functional - team has full set of skills, reduce handoffs, encourage autonomy
High frequency customer interaction - increase probability of delivering valuable software

After the training, we were on our way to continuous improvement as a team. The quality of life difference between the before and after was stark. Of course we still had a lot to learn but we now had the tools and the belief that we were on the right path. The move from waterfall to Scrum was a very clear win for us. Just like the industry invented new technologies to build software on the web it also needed to find new ways to organize the people doing the work.

To this point I haven’t mentioned the word “agile”. Scrum is considered an “agile” process. It emphasizes the build/inspect/adapt loop which is the most well-known, if over-simplified, way to describe the value of being agile. Agile places value on delivering functionality as soon as possible followed by close customer contact to see the impact of those changes, modifying the software in near real time in a continuous way. This is contrasted with waterfall, a linear process that emphasizes an extended analysis, build, test, and delivery phases.

Scrum helped us operate as a team and communicate better to each other. It gave us an operating system for the engineering function that felt robust and predictable vs the fragility of POW. The specificity of the Scrum ceremonies helped us by increasing the quality of our interactions. And it gave us a mechanism for improving how we did our jobs. But the big ticket item is actually delivering more value to customers and in that dimension I don’t think we saw a huge leap. First of all we didn’t really adopt all of the post POW items above. I’ll give you one concrete example. There were a lot of technology barriers to continuous delivery, especially in the 2000s. The effort to adapt the existing technology stack to delivering smaller features on a more rapid basis was significant. Both the technology available to us at the time and the knowledge to retrofit a large and complex product was very immature. We made progress but I can’t say we were agile. There is no agility if you can’t routinely ship high quality working software in short time frames. The existing technology and architecture was just too complex. We would need to chip away at it over years, which we did, but it was painstaking and often frustrating work.

I can go down the post POW list and make a similar case that we put in a good effort on each but not enough to be actually agile. However, we did follow the Scrum guide very closely and this dramatically improved the developer experience, even if we fell short of being really agile. Recall that “agile” means shipping small things often, getting feedback from customers, incorporating that feedback, then shipping more small things ad infinitum. An executive watching our Scrum adventure from a distance might conclude that we wasted our time and that agile didn’t live up to the hype. See what I did there? I said “agile” didn’t live up to the hype, not “Scrum”. We replaced our shabby waterfall process with a reasonable Scrum implementation but we didn’t get very close to being agile. But agile takes the hit in the eyes of observers.

It’s easy to conflate Scrum with Agile. Scrum is a concrete software development process. Agile is a philosophy best expressed in this set of (4) values and (12) principles that informed the authors of Scrum. Real agility requires fundamental organizational change, from the e-suite down. Engineering can follow the Scrum guide to the letter without bestowing agility on the organization. If the executive team doesn’t embrace and advocate for the 4 values of Agile, then nothing the engineering team does on its own will move the organization towards agility. Without leadership participation, there is a risk that Scrum can actually inhibit agility due to a number of hereditary vulnerabilities in the process itself:

Scrum’s immutability
Overpowered Product Owner role
Process surveillance tooling (Jira et al)
Sprints are an endless loop of micro-waterfalls
The story point fiasco
Development focused
“Commitment”
Ceremony fatigue

A note about the Scrum guide. It’s been revised many times over the years. The authors have been responsive to criticisms about some of the more controversial items contained in the guide. However, Scrum practitioners and training don’t consistently keep up with the latest guidance nor does the tooling.

In the interest of brevity I’ll take you through 4 of these 8 vulnerabilities. I’ll do the other 4 in the next sprint (haha).

Immutable Scrum:

From the Scrum guide: Scrum is free and offered in this guide. Scrum’s roles, artifacts, events, and rules are immutable and although implementing only parts of Scrum is possible, the result is not Scrum. Ok then…I guess we love empiricism for me but not for thee? It is a crippling paradox to be so closely associated with Agile while asserting that the process itself is a fixed concept. What’s contained in the Scrum guide is a very well informed methodology for software development. You can do a lot worse than simply following it exactly. But this kind of language encourages rigid thinking and the existence of Scrum cops on the team, folks who are very pedantic about the rules of the game. Self-organizing teams, if the phrase has any meaning at all, must be free and encouraged to choose how to work and that has to include the rules of the game itself.

OP Product Owner:

The role of the Product Owner is not impossible to perform well but based on its near mythical requirements it is very unlikely. To be successful, as defined in the Scrum guide, the PO must possess a combination of awesome skills: technical acumen, effective communication to the entire org and all customer segments, work with engineering teams in minute detail; must also be politically adept, skillfully navigating each level of authority in the organization; they are often required to be domain experts; and last, but definitely not least, they must be supernaturally sensitive listeners and keen observers of customer behavior. The Scrum guide abstracts all of this behind the buzz word “stakeholders”. I have worked with some very talented and excellent product people and none of them have been able to live up to this entire set of requirements. It’s simply not a reasonable job spec. It’s designed for failure and burn-out.

Surveillance:

Many teams fall into the trap of emphasizing data collection and analytics to report team performance to leadership. Ticket flows, burn downs, burn ups, code commits, story points completed, etc. Management often demands metrics and these software platforms deliver what the paying customer wants. The metrics tracked by these tools reveal little if anything about how much value the team is delivering to customers. It is easy for the metrics to become the sole focus of the team at the expense of the customer. The very presence of these features in tools such as Jira reinforces the notion that Scrum should be a metrics driven process and can be treated like any other business function.

Endless Waterfall:

The Scrum guide gives few details about what should happen inside of a sprint. The team pulls items into its sprint backlog and then “self-organizes” to complete the work within the sprint boundary. Sounds good so far but if a team has been doing POW, their first instinct will be to proceed with, you guessed it, POW. Let’s say the team pulled in 5 stories and the team is 5 people, usually each person will take one of the stories - this is how you end up with 5 unfinished stories at the end of the sprint. If the team routinely works one story per person per sprint continuously, you often end sprints with incomplete work. The team can find itself in a worse state than if they just stuck with waterfall, nearly incapacitated by this growing pile of unfinished work. It feels intuitive to waterfall inside of a sprint but it’s a trap, a never-ending loop of waterfalls. You will not have broken any of the immutable laws of Scrum if you do POW inside of your sprints but you will struggle to deliver value on a consistent basis to your customers.

Final thoughts:

Even though Bob’s reaction to Scrum was more instinct than experience, I had to admit that he got it right; Scrum had become, in Bob’s colorful phrase, corporate bullshit and I needed to understand how Scrum might have become the thing it was trying to fix. I found that Scrum itself is not especially bad. Despite this long post confirming many of Bob’s instincts, in the right hands Scrum - with some minor tweaks - enhances agility and can be a successful process for organizing software teams. The problem I see is that Scrum also works well in organizations that are determined to remain hierarchical and control oriented. The larger project of Agile has been hurt by its association with this kind of Scrum and has contributed to cynicism and doubt about the value of empirical processes for software development. Has the corporate takeover of Scrum killed Agile? The positive spin would be to acknowledge that Agile is deeply ingrained in many processes and organizations, however imperfectly and unconsciously. The darker view would be to conclude that Agile, unable to win over the e-suite, its grass roots origins co-opted by credentialism, consultants and hucksters, failed to deliver on the aspirations in the Agile Manifesto. I remain optimistic, however, that Agile and the manifesto is as relevant as ever. The pace of change will surely continue to accelerate, creating increasingly unfavorable conditions for rigid thinking.