The Trouble with Scrum

Why Broad-Brush Approaches to Project Management are Harmful

Project management methodologies are a bit like religion and politics: Everyone has an opinion, and those opinions tend to be strongly held.  This often creates situations where the framework is applied by force, whether or not it can possibly be effective, and the inconvenient or impossible-to-implement parts are ignored or dropped on the floor.

A High Level View

At this point, Scrum has a bit of a storied history; it has been around for over a decade, and has generated whole cottage industry of "official certifications," courses, books, consulting services, and products to help spread the good word.
It has a strong appeal to middle management: implementing Scrum creates a lot of motion - there are daily check-ins, tons of ceremonies, and measurable artifacts get produced. It provides a layer of perceived visibility into software engineering processes - which are often inherently opaque to non-technical project managers, outside of explicitly user-facing components. This can especially true when trying to reason about  more abstract things, like cleaning up technical debt.

It also has a strong association with Agile (which is no surprise, given that its creator was one of the Agile Manifesto Founders), and is probably its most popular implementation - this is  relevant because of Agile's general popularity, and frequent conflation with a "development methodology" - though it is really more of a set of concepts and guiding principles than a real "methodology" in and of itself.

The Good

While it is easy to poke at the "bad parts" of Scrum, I'd be remiss not to at least mention the things it does well:

• Stakeholder Engagement is something that (in theory) Scrum has done a great job of pushing.  Without this, the parts of projects that make them meaningful often get lost in the long list of hard requirements that must be delivered.
• Impediment Identification is also something Scrum dedicates substantial resources toward.
• Strict Timeboxing & Scope Management provides something that manages one of the biggest problems which tends to emerge with large projects: Scope creep, and ensuring too much time isn't spent on a particular feature through bounding to specific sprints.

Where Problems Begin to Emerge

At this point, even for those unfamiliar with Scrum, it's likely apparent that this will not about how Scrum is all sunshine and rainbows, and mass adoption will not solve all of your project management problems.  While there are some bits Scrum gets right (and does well), there are also a lot of sharp edges.

How Scrum is Optimized

The problem really starts when considering how Scrum is actually structured - it is a framework with lots of moving pieces, and specialized terminology. This seems to be somewhat at odds with the Agile Manifesto - which prefers Individuals and Interactions over Processes & Tools - but even ignoring that, it is a very opinionated framework.

In general, strong opinions may not be particularly bad, but one of the implications of being an opinionated framework is that a lot of assumptions are made about how work gets done. This means that the analogy falls down when approaching parts of a project, such as research-intensive tasks, as they may not bucket well into the way that Scrum views such tasks. Additionally, the way that Scrum manages scoping and executing tasks tends to almost optimize for the production of technical debt: Stories are scoped to a sprint, and developers are expected to deliver slices of functionality at the end of each sprint.

In theory, this makes it so that tracking progress across multiple sprints (in the pursuit of accomplishing epics) is much simpler, but it also makes it easier for projects to end up as a sprawling mess of half-finished features. It incentivizes development teams to take quick shortcuts to ensure that they deliver what they signed up for by sprint-end, and there is substantial complexity in managing the "definition of done" off of these quick snapshots of functionality.

Finally, Scrum also makes a number of assumptions about team structure: it assumes every participant is essentially cross-functional, and introduces several roles (i.e., Scrum Master) which often end up being either nebulously defined, or very underutilized in organizations. It also makes assumptions about organizational structure and adoption; this can make really implementing Scrum functionally impossible in many organizations, as it might not be possible to engage with the real stakeholders in the fashion Scrum dictates. Moreover, it is very easy for management to over-rotate on scrum ceremonies - this leads to large amounts of administrative overhead essentially being pushed down to the development team, which leads to not-insubstantial morale problems.

Ideal Use Cases

There are use cases that Scrum really excels at - consulting engagements, for example, or internal projects for a specific stakeholder. These generally follow a relatively predictable path, and are well-served by small, demo-able slices of functionality. Additionally, it is likely to be straightforward to keep the relevant stakeholders engaged.

The Dreaded "Scrum-But"

Scrum proponents would posit that almost all of these issues are not actually problems with Scrum itself, but rather, are scrum implementation problems. Either it is an issue with management's understanding of Scrum, or a lack of buy-in from senior leadership. The argument essentially being that they are using "Scrum... But" with important parts missing.

While there may be some truth in this, to a degree it seems more an application of the "No true Scotsman" fallacy: if most implementations have fundamental issues related to implementing Scrum, it seems likely that Scrum is (at least in part) probably the culprit. It is a very opinionated, prescriptive framework that requires fairly rigid cross-organization buy-in; and most implementations forget crucial parts of the process. Additionally, maintaining the required ceremonies without actively distracting engineering teams requires a level of discipline most organizations simply lack.

Summary

In summary, while I am not sure I'd say Scrum is bad per se, I have not seen a "healthy" implementation in practice; it is a very opinionated framework that requires very broad organizational buy-in. On top of this, most of its biggest proponents sell it as a panacea for project management - which it most decidedly is not. In fact, I would argue that it is actually a fairly inappropriate tool for many use cases, and often simpler Agile frameworks (such as Kanban) are much more appropriate.

While it is not without use cases, I would even go a step further, and posit that there is likely some additional incentive for a framework like Scrum to exist - the roles, complexity, ceremony, and pitfalls all make it difficult to implement holistically. This almost certainly opens the door to the entire "Scrum Industry" of consultants, books, videos, and training courses. It is a framework developed by consultants, after all.