It’s A Marathon, Not A Sprint

There is no single development, in either technology or management technique,
which by itself promises even one order-of-magnitude improvement within a decade
in productivity, in reliability, in simplicity.
– Fred Brooks, Jr

In the past few years, I’ve worked at various companies which used Sprints to plan their work. Agile methodologies, such as Scrum, have become the latest fads in software development. Every company I’ve interacted with touts how they switched to the Agile software development processes and it is the most frequently cited silver bullet for many establishments. There are many processes that the agile methodology advocates, but the one that is being adopted most rapidly is Scrum. You can find innumerable books, websites and blogs that will extol the benefits of Scrum, but any criticism of the process is really hard to find.

Scrum has many advantages similar to those of other iterative or rapid development processes. But due to the high popularity and the heavy marketing of Scrum as a panacea, very few people realize the potential drawbacks they could encounter. This post describes the various issues that I have observed in the companies that I worked at. I will not go into the details of how Scrum works, you can find many great resources on scrum websites. However, my favorite of post on agile is Steve Yegge’s sarcastic post on Good Agile, Bad Agile.

So what are the drawbacks of Scrum? The two symptoms I’ve seen commonly are:

1. Sub-optimal execution
2. Lack of solid software

Sub-optimal execution is any activity that results in rework or unnecessary wasted effort.  In this post I’ll concentrate on the this issue.  A follow-up blog post will delve into details of why Sprints may lead to poorer quality software. Let us look into some of the aspects of Scrum that could lead to sub-optimal execution:

No Longer-term Plan

Sprints are typically planned to be a 1-4 weeks long.  Many, in fact I would say most software development projects cannot be completed within this time frame, especially when they involve changing your core architecture or something deeply technological.  While sprints do a really good job of taking a fraction of the project and making sure it is completed in the allotted time period, what they lose out on is figuring out how the various pieces of the project fit together into a final deliverable. That aspect typically gets overlooked or mismanaged and people tend to forget tasks that are needed for the project to complete. Often these tasks are discovered either at the last possible moment or after deployment, and in both cases need some amount of scrambling and rework. The primary reason for things to fall off the plate is because there isn’t a well laid out plan and a person responsible for executing on it.

Another way the lack of a plan affects a team is when the team ends up consistently focusing on short term deliverables (week – 2 weeks) rather that coming up with a longer term (1-2 quarters) road map. This happens most often in teams with a large influx of high priority requirements. Sprint planning focuses on immediate delivery and that leads the team to deliver on requirements which provide immediate benefits rather than investing in longer term projects which don’t have an immediate payoff but will provide greater benefit a few sprints down the road. While this is not a drawback of the process itself, since a team can consciously avoid this pitfall, it happens as Scrum unconsciously forces people to think in a certain way.

Excessive Time Spent On Software Estimates

During each sprint a team loses up to 1-2 days per developer in planning. The big reason for this is that many teams spend a considerable amount of time on estimating how long it would take for them to complete the planned tasks. This tedious process involves taking a particular story (project) and breaking it down into bite sized (0.5 – 1 day) tasks and estimating them (typically in hours). Software estimates by their nature are almost always overly optimistic and especially so at a granular level. So spending several hours generating incorrect data and using that to determine an exact plan is just time ill-spent. Moreover, the team will often not finish what was planned, and so the additional effort that was spent during planning is wasted. An unfortunate side-effect of this process is that it causes unnecessary friction with management which believes that the team should always complete what it has promised.

As a workaround to this problem of optimistic estimates, people have added bizarre concepts such as ideal days. But ideal days are equally hard to measure and justify and more often than not estimates are still just as wrong. The problem is not with using estimates, almost all software processes use estimates. The problem is spending considerable amount of time on generating an extremely tight plan that leaves no room for the unexpected and is rarely what happens during the Sprint. Another complication that arises from this planning process is that if you err too much on the side of caution, it often leaves people with nothing planned to work on. This is more common near the end of the sprint if the tasks get over quicker than expected, or if people are blocked on certain tasks and have none other. Finding the right balance in the plan is incredibly difficult as it can vary widely from Sprint to Sprint and leads to sub-optimal use of time in either case.

Artificial Deadlines

Sprints are designed to finish all the planned work every few weeks. This results in recurring constant deadlines which are artificial and have very little direct correlation with external deadlines. Constant deadlines in themselves are problematic as they add unnecessary stress on the development team. But they also lead to poor decision-making since the deadlines are fixed in time and not related to how long it takes to deliver the projects. I have seen the following situation innumerable times during planning: A developer estimates a project to take 6 days, but there are only 5 more days left to plan in the Sprint. In a natural process the developer would just work on it and release the software a few days after the Sprint deadline. But not so in a Sprint, due to the constraint that a team should always finish what it has planned in the sprint, the planning process takes one of the following choices. Either the scope is reduced to finish in 5 days, or the project is substituted with another of lesser priority, or an additional person is assigned to the project. All of these decisions are sub-optimal and driven by the fact that the plan should always finish by the given date even though, a release a few days later would have made absolutely no difference to the stakeholders.

Other causes

There are a few other reasons which result in poor planning. One being the lack of continuity between sprints. There is little correlation between the work done in two consecutive sprints and projects rarely get planned correctly across Sprints. This can sometimes result in half done work left lying around or projects skipping sprints, requiring additional effort to restart on them. Another one is the requirement of homogeneity in the team. Scrum assumes that all developers are interchangeable. This is not true, especially in cross-functional teams – some people might be Javascript/CSS gurus, others might be backend developers, some may be QA engineers – each person in the team likely has a unique skill set. Asking people to work on aspects that they are not skilled at is sub-optimal, and planning work based on the composition of the team adds more difficulty to an already complex planning process.

Conclusion

Many of the issues I have outlined can be avoided by a team with some conscious effort. This leads people to argue that these are problems with the implementation and not the process itself. Any Agile Leader will tell you that a lot depends on how the Scrum process is implemented – and if implemented incorrectly it could be worse than the existing process. The first thing they ask when you express displeasure with the process is: Did you follow the process exactly as the book says? And when you say no, they would be quick to point out that it is exactly why the process failed for you. But of course no one follows the process exactly “by the book”. Each company that I’ve worked at had their own implementation based on their internal constraints. And while you can reasonably conceive of implementations of Scrum which do not these problems, the reason teams tend to fall in these traps is because the Scrum appears to promote a certain pattern of thought where it is natural to make these mistakes. Thinking differently and avoiding the pitfalls takes effort and the realization that there are better ways to structure your software development processes.

Teams are building the product for the long-haul and using Scrum is akin to a person running a marathon by sprinting for a few hundred meters and then resting and re-planning what they do next. There are very good reasons why marathon runners do not sprint until they have to at the very end of the race.