How to Succeed with Scrum

25/03/2011

I’ve been having an ongoing argument with a colleague who is critical of Scrum. He believes people choose Scrum because they wrongly believe it’s easier and cheaper than Waterfall, or because they have been sucked in by the current “Agile” fad. He says that people choose Scrum over Waterfall because they want to avoid the discipline of doing the hard work upfront, that they believe they won’t have to make difficult decisions and that they will save a buck or two along the way. I can’t say that he’s wrong in what motivates people to choose Scrum, but I do believe he’s mistaken that Scrum is easier and less disciplined than Waterfall. Succeeding with Scrum takes effort, discipline and many tough decisions along the way. It takes the right attitude and level of commitment, especially for product owners, to make it work. Choosing Scrum is not an easier path, but if you have the right elements in place, it can produce great results. The key is knowing what the right elements are.

Common Understanding

First and foremost, there needs to be a common understanding of how things are going to work. It’s not enough to say we are going to do “Scrum” without being more specific about the details of how Scrum is going to be implemented for the particular project at hand. This means taking time and effort upfront to be clear on a whole variety of things. Here’s an example of some of the aspects that need to be addressed before the project kicks off:

  • How much analysis, if any, will we do before our first sprint?
  • Does the team have to be sitting together?
  • At what time are we going to have our daily stand-ups?
  • Do we need to take meeting notes for each stand-up?
  • What happens if the product owner can’t make all of the stand-ups?
  • How long are our sprints going to be?
  • What’s our definition of complete?
  • Will we have sprint retrospectives?
  • Will we have pre-sprint planning?
  • What tools will we use to capture user stories?
  • What tools will we use to monitor progress?
  • What level of detail do we need for our conditions of satisfaction?

There’s no right or wrong answer for any of these questions; they depend on the specific project. It might be that it’s not possible to have the team sitting together as they aren’t in the same country. It might be that sprint retrospectives are not needed as the team has worked together before and it’s unlikely that they need a formal method of capturing the positives and negatives of a particular sprint, and that this will naturally come up as part of the daily stand-up and sprint planning.

The important point to note here is not what the answers are, but that the questions have been raised, considered and the team has come to a group decision. Nor is it vital that all of the questions are answered before the project starts; sometimes it’s fine for some questions to be answered at a later date. What’s important, however, is that the team agree that it can be left until later and that there’s a consensus on the way forward. In this respect, Scrum is much like Waterfall in needing to address elements upfront. It requires discipline to make important decisions. Where Scrum has the advantage over Waterfall is that the decision of how this project is going to be run is a collaborative effort. The key members of the team all have a say; it’s not the project manager dictating to both the client and the team how things are going to be run. It’s a vital first step to ensure the entire team works together in a way that they all agree to. This way they build consensus before the project starts.

Discipline

When the project starts is the time when the reality of what Scrum means comes into play. For people that are new to Scrum, it can be confronting and uncomfortable. It means putting aside preconceived notions of how things should work. This is where it becomes challenging for Product Owners that thought their lives were going to be easier being “Agile”. The reality is very different. Product Owners quickly find out how much work they have to put in for the project to become a success. In contrast to Waterfall, the Product Owner should be a part of the project everyday, they need to be actively involved in the analysis of features, daily scrums and sprint planning. This isn’t easy and requires discipline from the Product Owner.

Clarity

There are a number of qualities that a Product Owner needs for a Scrum project to be successful. One of the most important is the ability to be clear about what they want. This is no different from Waterfall, but it’s far more obvious in Scrum where the Product Owner can be disconnected from the project. In Waterfall, the Product Owner can wait until the specification is drafted, answer any business questions raised and have it handed over to the developers to build. Further questions are likely to arise but, unlike in Scrum, it’s not likely to be on a daily basis. In Scrum, there’s the chance that every day a developer will raise a question or blocker about a particular user story that hadn’t been considered before. This is when the Product Owner is accountable. Everyone on the team will be aware that the question has been raised and it needs to be answered for the project to continue. It puts pressure on the Product Owner to come up with a quick answer. Hence the importance of them needing to be clear about what they want. If they don’t, then that user story will have to be put on hold until it is resolved. The Product Owner has no choice but to accept responsibility for providing clarity.

Priorities

Another important quality for Product Owners is being able to prioritize. Very few projects have unlimited budgets and Scrum project are no different. Although the scope may have flexibility, it’s unlikely there will be an unlimited budget and endless resources. As those who have read Fred Brooks “Mythical Man Month” will know, having more resources on a project doesn’t necessarily make it go quicker and can, especially near the end of a project, have the opposite effect. What this means for the Product Owner is that they need to be able to prioritize the backlog and be able to identify what’s most important. When it’s clear that not everything in the backlog is achievable, the Product Owner has to make tough decisions about what is included and what gets dropped. This leads into the next quality: acceptance.

Acceptance

What is probably the hardest part of a Scrum project for a Product Owner is accepting the reality of what is achievable on a number of levels. Where this is most difficult is when the true velocity of a project becomes apparent. For example, at a sprint planning meeting, the developers estimate they should be able to complete 8 user stories in the next sprint. The Product Owner has to accept what the developers tell him about how long it’s going to take, even if they believe that it shouldn’t take that long. The Product Owner has to trust that the developers are telling him the truth about how long it will really take despite what the Product Owner thinks. That’s not to say that estimates can’t be challenged and adjusted, but ultimately the Product Owner has to show trust in the team. However, it doesn’t stop there. By the end of the sprint, unforeseen circumstances may have surfaced that meant only 5 user stories are actually completed. The Product Owner is unlikely to be happy about getting less than expected (who would), but, once again, they have to trust that the team has done everything they can and that the reasons for the delay aren’t their fault. Even more challenging, the error could be one of estimation, that the team underestimated the complexity and simply got it wrong. The Product Owner has little choice but to accept the reality and adjust the expectations of what can be delivered in the next sprint and possibly for the entire project, especially if the budget doesn’t stretch far enough for the entire backlog to be completed.

In contrast to Waterfall, it’s not known in detail upfront how much can actually be achieved for the number of sprints that the budget allows. That means the Product Owner has to accept that they may not get everything they want, and after each sprint, as the reality of how much effort is truly required for each user story, expectations may need to be adjusted yet again. Ironically, in a Waterfall project, the scope is clear upfront and the Product Owner can insist that it all be delivered, the risk being on the supplier and team to deliver, even if it takes more effort than originally estimated. In Scrum, the Product Owner has to accept both upfront and during the project that the scope of what will be delivered can and, more often than not, will change and they will get less than they hoped for initially. This is not an easy position for a Product Owner to be in, but the benefit they get is being able to choose what’s most important. Ultimately, for a Scrum project to be successful, the Product Owner has to trust the team and be pragmatic about the end result.

Commitment

Last but not least, the Product Owner has to show commitment. Unlike Waterfall, they must be there every day for the entire project. They need to be at daily stand-up meetings so they can answer questions and resolve blockers, they need to be at sprint retrospectives to understand how the team feels about how things are going, what’s working, what needs to be changed. They need to be available to participate in the analysis required for user stories to be tackled in the next sprint. They need to be at sprint planning meetings to prioritize the backlog. They need to be at showcases to see what’s been done and comment on whether it’s acceptable or if further refinement is required. It’s a huge commitment and vital for the project. Product Owners can’t simply tell a business analyst what they want, read the specification and then wait for the project to be delivered. They have to be committed for it to succeed.

Accountability

Clearly much rests on the Product Owner’s shoulders to make a Scrum project successful, but it’s not just the Product Owner that has challenges to face. For a developer used to Waterfall, it can be confronting to be so exposed, to not be hidden behind the project manager, and be accountable for estimates and their progress on a day to day basis, and to deal with the change that Scrum allows.

The first thing that a developer will notice about Scrum is the direct contact with the client, or in this case Product Owner. I’ve worked on projects where the key technical person has had little or no contact with the client. The majority of the communication has been via a business analyst, sometimes from the client’s business analyst to the supplier’s business analyst and then the lead developer. In Scrum, not just the lead developer, but all developers get to talk with the Product Owner every day. For some developers, that’s an uncomfortable situation. They may struggle with communicating technical challenges in layman’s terms and succinctly convey blockers, issues, concerns or solutions. In Scrum, developers can’t hide behind technical terms and sit in their cubicles or offices; they have to step outside of their comfort zone and face business people that will ask simple questions that may not have simple answers.

Transparency

Along with having to deal with the Product Owner on a daily basis, the developers’ progress will be monitored on a daily basis. Developers aren’t left to their own devices for days or weeks at a time, they need to break down the user story into individual tasks that are assigned story points and they are measured, daily, against their estimates. If they have forgotten a task, they add it in and everyone will know they forgot. If they take longer on a task than they thought, then everyone knows. That’s not to say developers will be blamed for missing tasks or taking longer than expected, we are all human and don’t always get it right. The difference in Scrum is that everyone will know, the message is not delivered by a Project Manager where the developer has never even met the client. In Scrum, the developers’ estimates and progress are totally exposed. The fact that it might be accepted and understood by the team still doesn’t make it easy to admit that you were wrong, and for some developers that’s a very uncomfortable situation to be in.

Adaptability

Although developers are exposed in the project, the challenge of Scrum is that they are not exposed to the details of the entire project at the start. And that’s because the details don’t necessarily exist. There is usually a backlog of user stories, but that doesn’t mean the backlog is what will be developed. Some user stories can be dropped, some can be added, some can change during the project. It means there’s a lot of flexibility to ensure the end result is what the Product Owner both wants and needs. The challenge for developers is learning how to be flexible, how to adapt to change during the project, how to technically design a solution when you don’t know all the moving parts upfront. This is one of the drawbacks of Agile. Without knowing all the details upfront, it’s very difficult to design a holistic solution. Sometimes the developer has to implement a solution and then later, when a new user story is introduced or existing functionality is changed because the Product Owner has changed their mind, has to refactor code already written and tested or introduce work-arounds due to other design decisions made before the change was identified. This can be incredibly frustrating for intelligent developers who take great care to design efficient, robust and maintainable code.

The irony of the challenges facing both Product Owners and developers is that it is exactly those aspects of Scrum that make it so successful. So many projects suffer from lack of involvement from the client. In Scrum, the Product Owner has to be committed and a part of the project on a daily basis; they have to be there to resolve blockers, make decisions on priorities, clarify details, determine the conditions for success. For Scrum to work, and for any project to have a chance of success, the Product Owner has to be involved, has to be committed and has to know what they want. Similarly, many projects suffer from the developer being too far removed from the project, not being accountable or having to explain and justify their actions. In Scrum, developers can’t hide, they are held accountable on a daily basis and have to be clear about what they are doing, when it’s going to get done and justify why it might take longer than expected. Scrum ensures that there’s transparency which leads to a greater chance of success.

In summary, what’s required for Scrum to work is what’s required for any project to work.There needs to be a team where everyone is clear on the goal, is accountable and works together to deliver the end result.