Why Agile Works


Agile works and it works well. But so can Waterfall, RUP and home grown methodologies as well as having no particular method at all. And they can all fail. The question is: Why does Agile work? What is it about Agile that is different? What is it about Agile that provides a better chance of being in the 32% of projects that actually succeed? And, most importantly, can it work for you?

What is Agile?

If you’re willing to embrace Agile, it pays to understand what it is, whether it’s right for you and how to avoid the common mistakes that lead to Agile adoption horror stories. Let’s start with what the term “Agile” actually means. In my experience, when people use the term “Agile”, they are either referring to a particular Agile methodology, such as Scrum, or to particular techniques used in Agile methodologies, e.g., daily stand-ups, user stories or pair programming. I’ve also heard people refer to “agile development” without necessarily elaborating on what exactly they mean by that. However, in general it seems that, more often than not, people most likely refer to a “scrum-like” methodology when they use the term “Agile”. Of course, that’s somewhat misleading. The origin of the term “Agile” comes from the Agile Manifesto that defines four values and twelve principles. The methodologies that are considered Agile are particular methodologies that attempt to, for better or worse, embody the values and principles set out in the Agile Manifesto. The first and most common mistake I find is that people think Agile means Scrum or XP or FDD (Feature Driven Development). Well, it doesn’t. Separating out the term Agile from Agile Methodologies is the first and probably the most important step in understanding and becoming Agile.

Let’s look at a few of the principles of the Agile Manifesto and compare them to the practices defined in Scrum and FDD.

  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Scrum has the concepts of sprints, typically a 2 to 4 week period in which the team commits to building a deliverable working product. FDD has the concept of feature sets, and features. A feature is between 2 hours and 2 weeks of work.

  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

In Scrum, at the end of a sprint, there’s a sprint review meeting where the stakeholder gets to view the working product and is able to make changes to it which is then added to a future sprint. In FDD, when a feature set is delivered, it is demonstrated to the stakeholder. If during that demo changes are required, they are captured as new features to the existing feature set.

  • Business people and developers must work together daily throughout the project.

In Scrum, there are daily meetings called “daily stand-ups” where all core roles, Product Owner (stakeholder or client), the team (e.g., developers) and the Scrum Master (similar to a traditional Project Manager) are present. In FDD, before development begins on a feature, the domain expert (often a key stakeholder) walks the team through where the feature fits into the overall solution.

What’s important to understand is that Agile is actually about values and principles. Agile methodologies are particular ways of implementing those principles. Some agile methodologies deal with all of the 12 principles, some deal with only a subset. An approach taken by Scrum might work well in one situation and poorly in another, the same with approaches taken by FDD or XP. Once we understand the distinction between Agile and agile methodologies, we are free to choose the method that is right rather than assuming Agile means Scrum and therefore to be “Agile” we have to do what Scrum says, even if we don’t agree with it or it doesn’t make sense in our particular situation.

What is it about Agile that makes it work?

Now that we understand what Agile means, the next thing to understand is why it works. There are a number of reasons, but one stands out above all, and that is that it’s focused on delivery. Seems obvious, doesn’t it? After all, the main reason for a methodology is to help you deliver a result. right? Wrong. That’s the key difference between Agile and traditional methodologies. If you look at Agile, the values are the following:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

The Agile Manifesto recognizes that while there is value in processes, documentation and plans, there’s greater value in individuals, and in working software.

If you look at Prince2, the defacto standard methodology in the UK, its key features according to its website are:

  • Focus on business justification
  • A defined organization structure for the project management team
  • Product-based planning approach
  • Emphasis on dividing the project into manageable and controllable stages
  • Flexibility to be applied at a level appropriate to the project. 

Not once does it mention actually delivering an end result. That’s not to say Prince2 is not a good way to manage projects, it certainly has its strengths, but a focus on delivery is not one of them, its focus is on control. Agile states “Working software is the primary measure of progress”, Prince2 focuses on business justification. If the problem you want to solve is not being able to deliver on time and budget, it’s pretty obvious which path to take.

This distinction between delivery versus control deserves further scrutiny. The Chaos Report states that for all projects in 2009 “44% were challenged which are late, over budget, and/or with less than the required features and functions and 24% failed which are cancelled prior to completion or delivered and never used.” Controlling a project that is late, over budget or with less features than required, let alone being cancelled, is somewhat pointless. Delivering working software is the point. Agile doesn’t state specifically how to do that (Agile methodologies do), but does define the principles that will lead to results. That’s the key distinction between traditional and agile methodologies: the focus on delivery.

After delivering, the next main reason why Agile works is the recognition of the value of collaboration. This is well illustrated in the practice of daily stand-ups in Scrum. Every day, the core members of the team meet. I’ve seen projects where there’s been 3 months between the client signing off the specification and them seeing a piece of working software, and in that time, they had forgotten half of what they had asked for in the first place. Whether it’s FDD, Scrum or XP, collaboration is key. The principle of Agile that best captures this is “Business people and developers must work together daily throughout the project.” Why is this so important? For me it’s to ensure everyone is on the same page. It avoids what I call the “over the fence” mentality. That’s where one person fulfills their part of the project, e.g., writing the business case, writing the requirements, writing the functional specification and hands it over the fence to the next person who does the technical analysis, does the development, does the testing...etc. Each person takes responsibility for their part of the process, but just that and only that. They pass it over the fence to the next person and it becomes their problem. For example, if an issue arises with a particular element of the functional specification, the business analyst may have moved on and it’s up to the developer to resolve this issue. If they get it wrong, they blame the business analyst who may or may not be around. And more importantly, the key stakeholder may only find out months later. It’s the “not my fault” syndrome and one thing you can be sure of, it’s NOT going to be the stakeholder’s fault. Agile ensures the risk of this is minimized by having regular interaction on a daily basis. Issues surface earlier and are easier to solve because business people are involved during the process.

Knowing the problem

Delivery and collaboration are the key to what makes Agile work. Making Agile work for you, however, isn’t as simple as saying “let’s go Agile”. There’s a bit more to it than that. As John Dewey said, a problem well stated is a problem half solved. Clearly understanding the challenges you are facing in delivering projects is the first step. Based on this understanding, you are well placed to evaluate the different agile methodologies and practices to determine which one is best suited to solve your problems. It might be the case that your challenges aren’t with the current methodology, but that there’s actually a people problem or management setting unrealistic deadlines. Using an agile approach isn’t going to make the impossible possible. If the fundamental issue is that timeframes and resources aren’t sufficient to deliver the project, Agile isn’t going to solve the problem. It might make it less worse, but it’s not going to change a fundamentally flawed project. As Brooks stated long ago, there is no silver bullet and Agile is no exception.

Picking the right agile methodology

If you are clear on what the key issues are, the next step is to pick the right agile methodology for your situation. A good place to start is looking at the Cockburn scale. As not all projects are alike, so too not all agile methodologies are alike. The scale and nature of a project will determine how formal an approach is required and which agile methodology is best suited. Cockburn looks at projects in terms of size, i.e the number of people, and criticality, i.e. the impact if something goes wrong, will there be loss of life (e.g. software for car safety) versus loss of comfort (e.g, the search feature on a website is not working). There’s a big difference between a project with a team of 100 working on anti-lock braking software versus a team of three working on a website for a new soft drink. The following figure takes the Cockburn scale and overlays the different agile methodologies and where they are best suited.

As the diagram indicates, XP works better with smaller projects, FDD with medium to large and Scrum applies across the board. That being said, don’t forget that just because this diagram indicates that Scrum can work for all size and importance of projects, doesn’t make it right for your particular situation. If you deal with projects that are larger in nature, it simply indicates that XP may not be best suited.

Implementing Agile

If the first step is to understand Agile, the second to understand your challenges, and the third to select the methodology best suited to your particular situation, the fourth and most difficult is to introduce the new methodology into your organization. The difficulty of introducing change should not be underestimated as Machiavelli observed many centuries ago.

"It ought to be remembered that there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things."

Change is difficult, scary and unsettling, but without change there can be no improvement. There are so many challenges in introducing change that entire volumes have been dedicated to the subject. Change management is an entire discipline in itself and cannot be easily summarized. The best approach to change will depend on the particular situation just as the best agile methodology will depend on the particular types of projects.

In my own experience, I faced two major challenges. It was back in 1999, before the Agile Manifesto was even written. I was Development Manager for a large web development firm (over 150 staff). There was a general agreement that we needed to do better, our record in project delivery was as bad if not worse than the industry standard set by the Standish Group8 at the time. The development team had evaluated a number of methodologies and found one they wanted to use, FDD. It was up to me to convince the rest of the organization that we should give it a try. That was a lot harder than it seemed. Because I had, somewhat foolishly in hindsight, done the evaluation of methodologies as a part of the development team only, getting buying from the Business Analysts and, more importantly, from the Project Managers was difficult. It took a series of meetings to even get acceptance to give it a go on a pilot project. But I didn’t want to fall into the trap that many people make do when trying something new, I wanted to get training first. This meant spending money, not a small sum of money either, as I needed to make sure I had the key developers, business analysts and project managers at the training to ensure we all knew what we were doing. Funding was nigh impossible and I had to raise the funding myself (out of a previous business I had owned) to get started on the understanding I would be paid back by my employer at the next quarter. Fortunately my punt paid off (yes, I was paid back) and we had a cross-discipline team who understood how our chosen methodology worked. This lead to a pilot project and over time we got better at project delivery, slowly but surely.

An important point to note is that after our training, we realized that FDD wasn’t going to work perfectly for us, we would need to make some changes to the methodology to adapt it to our environment. As a part of a group workshop, we decided what elements we would adopt and what we would adapt and what we would leave out. Being selective about what we took from FDD helped us to ensure it would work. I would be lying if I said that we applied this adapted approach to all projects, but the underlying mindset was definitely in the forefront when we tackled each project, and because of that we did better as a whole. It was more of an evolution rather and an overnight revolution. Change takes time, expecting to get instant results because you’ve decided to go Agile is only going to lead to disappointment.

Another aspect of change to keep in mind is that it’s rare to try something new and get it right the first time. Learning new skills takes time and persistence, making mistakes and learning from them is a part of the process. Trying an agile methodology for the first time may lead to worse results than the usual approach as people learn how the methodology works in practice. Ideally there’s both training beforehand and someone with experience on hand when first trying to implement an agile methodology to help guide you through the challenges that will inevitably arise when first trying something new. Setting expectations upfront that it’s a learning experience will help to avoid the backlash when it doesn’t work perfectly and prevent the natural inclination to revert to a known approach.

To sum up, Agile works because it focuses on what matters, delivery and collaboration. Making it work is the hard part. The first step is to understand the issues you are currently facing and seeing if Agile, whether it’s a particular methodology or a set of Agile practices will help you solve the issues at hand. If so, then you can look at how to move towards an Agile approach, preferably with expert support, before, during and after to help make the transition. Knowing what the problem is and how you intend to solve it helps sets more realistic expectations and a greater chance of Agile helping. Blindly adopting an Agile methodology and assuming it’s the answer to all your problems is mistake. Agile does work, but it’s upto you to make it work.