Successful Web Development Methodologies
Commercial web development has been around for more than 10 years. As an industry, we are still fairly young when you consider others that have been around for centuries. But our relative youth as an industry is no excuse for not doing better. From the number of sites that have been rebuilt for clients, it’s clear that there is still a lot of poor quality work being done and this affects us all. It means clients are more wary and less trustful of web developers. Anything that tarnishes our industry can tarnish all of us individually. This leads to the obvious question – what’s the answer? Having tried, trusted and standard approach would go a long way to helping avoid the mistakes we see over and over again. Ie. a web methodology. However, finding a methodology that works for web development is not easy and making it work is even harder.
Back in the heady dotcom days as the Development Manager for a team of 20, this was exactly the dilemma I faced. This article covers the issues that arose and how we as a team went about trying to resolve them. What came out of this was the successful adaptation of an existing methodology for web development.
There were a number of factors that combined to force me to make a change in the way we did things.
First and foremost, projects were constantly going over time. There wasn’t a specific reason for this, each project seemed to have it’s own particular issues. For some it was the client changing their minds, for some it was us interpreting the requirements differently to what the client expected, for others it was a simple underestimation of the work required. Regardless of the reason, the end result was the same. Projects that go over time go over budget unless the client is willing to pay more. It also impacts new projects.
The more this happened, the worse things got until we had a situation where almost every project became high risk and suffered from scope creep, low morale and unrealistic deadlines.
The answer was pretty simple, we needed a better way of doing things, ie. proven way of delivering projects on time, on budget that met client’s needs AND would fit in with our culture.
We needed a methodology.
Adopt, Adapt or Build Your Own
Once the decision had been made to find a better way of doing things, there were three paths we could take.
adopt an existing methodology
adapt from an existing methodology
build our own
The development team were divided on this issue. Some believe we should make it up ourselves, others said we should avoid re-inventing the wheel. It was clear we had to do some research to work out which was the best path for us.
At the time, there were no recognised web methodologies (although from recent research the situation hasn’t changed much). So, for adopting or adapting, we had little choice but to look at the existing software methodologies
When we started looking for methodologies, we decided it was important to know what we were looking for so we decided on the criteria by which we would evaluate them.
It had to be more than just a simple guide, but at the other end of the scale, if it was too big, there’s no way it would work. Something that was easy to understand on the surface, client’s had to be able to understand it, but had the depth required to give developers the guidance they needed.
It’s hard to get people to read documentation at the best of times so a thick folder full of documentation was unlikely to be effective. Chances were that if there was a 100 page guide with a 10 page summary at the end, most people would only use that.
Anything that cost money would have to be justified, the less money required, the better.
We also couldn’t afford to get it wrong. It was unlikely that I’d be able to convince people to go through the process of trying a second methodology if the first one didn’t work.
Simple – it had to actually work, not be based on theory. There had to be real world examples of where it had been applied successfully, more than once!
Rational Unified Process
We had two presentations from RUP representatives. The first was a one hour session, the second a more detailed two hour presentation. Both times, I was more confused after the presentation than before. I also spoke to a number of friends that had worked with RUP and had positive things to say about the process.
The scope of RUP is extremely broad. It covers pretty much everything in the entire SDLC. This is both it’s strength and weakness. I was recently at a conference and attended a session by Phillipe Krutchen, one of the leading authorities on RUP. His view was that the main problem that arose when people tried using RUP is that they tried to use all of it. This is was the same advice that I got from my friends that had used RUP. It is large and sophisticated, the key is to use only the aspects of the process that you need for your project. This makes a lot of sense given that projects often vary and the same approach doesn’t always work.
However, given the context of our team, there were a number of issues with RUP.
it is large, complex and sophisticated, our team was not, we had some people that were still learning about the SDLC and trying to introduce something as RUP would be extremely difficult and require lots of training
even though RUP is comprehensive, we would have to review it in depth to then decide which elements we would apply to our projects, trial and refine over time
the cost associated with the tools that were required to use RUP (eg. rational rose, requisite pro…etc) was high
Overall, to implement RUP, or a scaled down version of RUP would have been a complex, difficult, time consuming and expensive process that had a high risk of failure.
We also got a presentation from a Process Mentor representative. The process itself was more compact than RUP and therefore easier to get one’s head around. In essence it was a website with a series of steps, forms and templates that could be used to run a project. We felt more comfortable with Process Mentor than RUP because is less overwhelming but it still wasn’t quite right. There wasn’t anything that stood out as a major issue as happened with RUP, but nor did it feel like the right approach.
In House Methodologies
As a part of the team, there were a number of experienced developers that had worked with “home grown” processes from previous jobs at a range of companies including heavyweights such as IBM GSA. We had each of these developers talk about their experiences, what worked, what they liked, what they’d use again. There were a number of techniques that sounded useful but the overall feeling was we weren’t going to be able to “borrow” one of these in house methodologies.
Why traditional methodologies didn’t fit
There was no shortage of vendors out there willing to tout their processes and associated tools. After many presentations, we didn’t feel any the wiser. Nothing seemed to address our needs. The reasons varied but the underlying problem was that none of the methodologies took into consideration the way things worked in web development. In comparison to traditional software development time frames are often shorter, the experience levels of employees vary dramatically, clients often have a poor understanding of what’s possible, the technology changes rapidly and it’s all via a single user interface (the browser). That’s not to say these elements don’t exist in traditional software development, however they are much more pronounced in web development.
Another issue with traditional methodologies is they they failed to take into consideration the “soft” aspects of software development. A recent study (Cutter Journal March 2005 – will get reference) of the most important factors in successful software teams ranked trust as the number one factor and technical expertise as last out of 17 factors. No where in any of the presentations or literature for both RUP or Process Mentor was trust mentioned, or any of the other soft skills that have a huge impact on success.
So, the short answer was we couldn’t either adopt or adapt a traditional approach leaving us with the unenviable task of trying to create our own.
We were about to start defining our own process when I came across a lightweight methodology or what is now called the “agile” movement. The methodology in this case was Feature Driven Development (FDD). Some of the other popular agile methodologies are XP, Scrum, Crystal and DSDM.
It was pretty clear that FDD was far better suited to web development than anything else we had seen and we decided to investigate further to see if we could adopt or at least adapt it to web development. It didn’t take long for the team to agree to give FDD a go. However, like most things, we realised it wasn’t a silver bullet.
PROCESS 1 – DEVELOP AND OVERALL MODEL
A initial project-wide activity with domain and development members under the guidance of an experienced object modeller in the role of Chief Architect.
Process 1 involves the project team creating an object model of the business domain that is more shape than content. The model is not fully defined with all attributes and methods, it is more about getting the shape of the business domain captured correctly in an object model, not capturing every detail.
The process is highly collaborative process where everyone has to work together to create the overall model. It takes an iterative approach where the domain expert explains a part of the business domain. The project team is broken up into groups (preferably 3 per group) to model that part of the domain. The groups then present their models and consensus is reached on which to use. This is then repeated for each part of the domain until everything has been covered. The end result is an overall model of the entire domain.
PROCESS 2 – BUILD A FEATURES LIST
An initial project-wide activity to identify all the features to support the requirements.
Creating the feature list is the task of the Chief Programmers involved in the modelling process. The Client and Stakeholders don’t need to be a part of this process as they have made their contribution in Process 1 and now it is a matter of capturing the project in a list of features. This does not require collaboration, getting a group of people involved at this stage would not be productive or constructive. The key to this process is defining the project using the language of the business domain. This means that the Client will be able to understand and value each feature. It also enforces a common language across the project team and reduces the risk of miscommunication or assumptions.
Poor communication is the basis of most problems in software development. The language we choose has a significant impact on how effectively we communicate. There are many techniques in FDD that help to provide meaningful communication, the most powerful one is in Process 2, defining the entire project in features using the language of the domain, i.e. using the Client’s language. This might sound simple and obvious but should not be underestimated. The focus that this brings is incredible and affects the project in many ways.
In FDD, everything is described as a feature which is defined as
the by/for/to an
calculate the total for a sale
calculate the total sales for a cashier
A feature is also defined in terms of size, ie. more than 2 hours work but less than 2 weeks work. If it more than 2 weeks work, then it should be broken down in separate features.
Having everything defined as a feature avoids the problems that occur all the time when the Client refers to something in one way and the programmer refers to it in another and the Project Manager has to continually interpret. If the Project Manager doesn’t get it exactly right, mistakes happen, the programmer thinks they are building one thing, the Client thinks another. Using the same language doesn’t mean the problem disappears, but it reduces the risk considerably.
PROCESS 3 – PLANNING
An initial project-wide activity to produce the development plan.
This process extends the benefits provided by Process 2. It provides the Project Manager with a way of planning the development phase in a meaningful way for both the client and the programmers. It is done in conjunction with the Development Manager and Chief Programmers. They look in particular at the order in which features will be built, balancing load within the team and strategies delivering early results to keep the client happy.
PROCESS 4 – DESIGN BY FEATURE
A per-feature activity to produce the feature design package.
Process 4 is broken down into three steps, walkthrough, design and inspection.
In the walkthrough, programmer familiarises themselves with what they are about to build before starting on a detailed design which is then inspected before they can start building. The inspection of the design means defects can be found and removed before a single line of code is written for that feature.
It might seem like commonsense to design and inspect before building but it is often ignored. In many other industries, the idea of building something before it has been fully defined, designed and planned would be considered negligent, yet it happens all the time in software development. The first reaction of many programmers, especially in web development is to open their favourite editor and start coding. The amount of risk that this approach exposes to any project to is enormous. However the converse can also happen where there is an attempt to design everything upfront and can often result in “analysis paralysis”.
Exactly how much design should occur upfront is a hotly debated topic amongst advocates of agile methodologies. The approach taken in FDD is seen in the difference between Process 1 and Process4. As we have seen, Process 1 includes design early in the lifecycle, but it is not a detailed design. The detail is left to Process 4. Putting the detailed design in Process 4 ensures that it is considered at the right time – before the code is written. It also breaks the design down into meaningful chunks, feature by feature. This means programmers don’t feel like they are spending all of their time designing and no time coding; immediately after the design has been completed and inspected, the programmer can start coding.
PROCESS 5 - BUILD BY FEATURE
A per-feature activity to produce a completed client-valued function (feature).
Process 5 is also broken down into three steps: code, code inspection, and promote to build. As with Process 4, the idea of collaboration and benefits inspections is enforced. What makes Process 5 unique is the final step “promote to build”.
For code to be “promoted to build” it must be finished. The key to this is the definition of “finished”. A feature is not finished until there is nothing else to be done. The way a Project Manager can test if a feature is truly finished is simply by asking, if the programmer answers “yes”, the Project Manager then asks “so there’s nothing else to be done?”. This often gets a different answer. It’s not that the programmer is being difficult or misleading it’s just that given the chance, many programmers will continue to work on code, tweaking, optimising or trying to improve it. If there’s time to do this then it’s not a problem, but if there’s a tight deadline, the Project Manager needs to focus programmers on getting the project completed and this is a great way to ensure that focus.
The other benefit of this Process is that helps the Project Manager see clearly how much of the project has been completed and how productive each programmer is. According to Gerald Weinberg (Quality Software Management Vol.1), the productivity difference between programmers can as much as 20 to 1. The issue is how to assess the productivity. Even though features might range in size from 2 hours to 2 weeks of work, it doesn’t take long for the Project Manager to assess how much work each programmer is actually producing once the variances in feature size are taken into account.
Implementing a new way of doing things is much harder than it sounds. People don’t like change. Even people that say they are willing to give it a go can quickly change their minds and revert back to the old way of doing things. Because of this, we knew that we as a development team would only get so far trying to apply FDD ourselves. Sooner or later, it was going to affect the project managers and eventually clients. We decided it was better to get everyone involved upfront to increase our chances.
The next step was all about internal politics and getting the key decision makers on side. As FDD is simple to understand, it wasn’t difficult for people to see that we would be better off if we took the same approach. Based on this, we got the funding to run a training course. Attending the course were representatives of all the groups who would be affected by a new methodology. Included were developers, business analysts, project managers, a designer and a tester.
At the end of the course, everyone was keen. In particular, the following aspects of FDD stood out and provided solutions for the symptoms we were suffering:
But we also learnt that it wasn’t going to be the complete answer. FDD covered development well but didn’t address gathering requirements, interface design or testing. The things that concerned us about applying FDD were as follows
High reliance on Tech Leads
Doesn’t deal with User Interface design & build
Not as powerful on smaller projects
Doesn’t cover testing & deployment
Not a Silver Bullet!
In the end, we were going to have to adapt the process to our situation and needs.
Managing the Transition
The next step was working out how to actually apply FDD to our work. There were many projects in production and we couldn’t simply change our approach midstream. Nor could we expect all new projects to start using the new methodology as not everyone had training. We realised we were going to have to take a staged approach.
We decided to tackle the problem from a number of angles. Firstly, we were going to slowly introduce a few of the key aspects of FDD into new projects. These were
Define projects using features (customer focused)
Plan development based on features
Implement new team structure, design & code reviews
Conduct weekly project status meetings
Secondly, we were going to run an internal test project. Our intranet needed redoing and we thought it was a good opportunity to give FDD a go as well as trying to work out how to deal with the interface and testing areas not covered. Thirdly, although not planned, a developer and project manager that had been through the training decided to use FDD on their next project for a paper merchant.
The intranet project didn’t go well at all. However this was for many reasons, none of which had anything to do with FDD. The project suffered from the same problems as many internal projects, it lacked buy in and a dedicated stakeholder. Even though we went through a fairly comprehensive requirements gathering process, the project lacked a clarity of purpose. It didn’t matter what development process was going to be applied, the project was going to be a difficult one. But what it did indicate is that FDD works best when there are clear requirements to start with.
The paper merchant project was a different story. It went extremely well. The developer, one of our best, did an excellent job in modelling the domain and the project manager was able to easily track the project. There were some challenges though. In defining the overall model, FDD uses a modelling technique called colour modelling (Java Modeling In Color With UML - http://www.amazon.com/exec/obidos/tg/detail/-/013011510X/104-3565645-1171160). Normally modelling is not done in collaboration with the client, in FDD is it and therefore the client needs to have an understanding of it. That doesn’t mean they need to be UML experts but at least have an idea of what’s happening. Explaining this to the client was the only real challenge and with most clients, this is a fairly straight forward and once they get it, they actually find it fun!
As the project had a small project team, one developer, one designer and one project manager it went smoothly. There was no need to create work packages, assign features to individuals, track progress of each individual…etc. One of the issues we had with FDD in the post training workshop was the reliance on the chief architect. We had only one developer so he played chief architect, chief programmer and developer. This was risky as if he was not good at his job, the project would be in big trouble – however this is the case with all small projects, if you don’t have a decent programmer, you’re in trouble!
Overall, there was a significant improvement on a number of levels, some of which were quite unexpected. The goal was to find a way to deliver project more effectively. What we found was the effect on morale was much more positive. The simple fact that we had decided to improve the way we did things had a positive effect. The team were happier because we were doing something about it. It also helped to bring together the different disciplines (development & project management in particular) who had previously not always worked that closely together. There was a palpable sense of focus and direction.
In terms of projects, there was also an improvement. Although most projects still had issues, they were lessened and easier to manage. It was much easier to get a real idea of where the project was at, how much had actually been completed. For instance, asking a programmer how they were going with database optimisation can lead to one of many answers that doesn’t necessarily answer the question that the client wants to know, ie. when will it be finished! That’s not to say that programmers deliberately try to avoid answering questions it’s just that sometimes the questions are hard to answer, sometimes it’s not that easy to say when something is finished. This is a key benefit of the feature driven approach. When defined in features, it removes some of the complexity in the questions being asked. A feature should be no more than two weeks work, so when asked how long before a feature is going to be finished (assuming it has started!) the answer should always be between 1 day and 2 weeks. It actually helps developers manage themselves and avoid trying to make the project manager happy by telling them what they want to hear rather than what the truth is.
Unfortunately, we were unable to monitor the long term impact of the introduction of FDD as within 6 months of the training, the dot com collapse hit the company and we went through a number forced redundancies.
FDD for Small teams
One of the main criticisms of agile methodologies is that they don’t scale up. For web development, it’s more important to see if it can scale down and still retain it’s advantages. With any methodology, it’s impact will decrease as the team size drops. FDD is no different. However, I have been able to successfully apply FDD to web development projects to small teams and projects, eg. 3 people on a 3 week project (project manager, developer, designer). FDD can scale down and still provide value to the web development process.
The two aspects of FDD that are of most value in small projects are
defining the project in features
tracking the project by features
It seems very simplistic but it is that simplicity that makes it so effective. Fail to define the project upfront in features and chances are the client and team will be on a different page from day one and it will only reveal itself once work is delivered. For small projects, correcting such errors is not a big task, however, a few days extra work on a small project like this is considerable in terms of percentage and can quickly eat into the profit margin.
Like most things in life, get started on the right foot and things are much more likely to work out well. Using the simple technique of defining the project in features using the client’s language will make a big difference to small and large projects alike.
FDD for Web Development
Since first applying FDD to web development in 2000, I have worked on refining the process and have come up with an approach that has worked effectively on dozens of web projects ranging in size from two weeks to 6 months. It uses the core of FDD and introduces new elements to manage some of the areas that FDD doesn’t cover. Below is a high level overview of how FDD can be successfully applied to web development.
Although it is not explicity stated in the 5 processes of FDD, it is an important part of every FDD project to have clarity of purpose. A simple statement defining what the project is and what it is supposed to achieve.
This would seem like a redundant element of a project, it should be clear what the organisation’s purpose is but it although it might be obvious to the client, it might not be obvious to the developer. Getting down a single paragraph of why the organisation exists is an important step. Ofcourse it is easier said than done, for smaller clients, the purpose is not always clear and asking this question of a client can sometimes bring about a very blank look! Still, it is well worth the effort to ask, it can help in the next step of working out the purpose of the project.
Once again, a very simple statement that is much harder to define that it sounds. For most clients, there are many thoughts, wishes, expectations and desires wrapped up in what they describe as their “website”. In reality, it needs to be defined as a project with a clear purpose that everyone understands and agrees to. It should be a clear, concise and measureable statement of the business outcomes the project is supposed to achieve.
The specific objectives of the project. A series of bullet points is fine, the act of thinking it through and getting the client to agree is the most important aspect of this part of the project overview.
Eg. the following objectives is for a car parts manufacturer
The objectives behind the redevelopment are as follows:
Unify ACME's corporate image worldwide.
Build a commerciallyorientedwebsite projecting a 'simple, sharp, professional, advanced and confident tone and one which puts user-friendliness first.
Ensure the design of the website adheres to the ACME Group Website Basic Guide Version 1.0
Use the website as a positioning tool to position ACMO as a technology-drivenleading global supplier of automotive parts, systems and components.
Ability for ACME to update the site in-house using a content management system (CMS)
Understanding and defining the project scope is a challenging task. Once again, clients usually want to include everything and expect everything to be done for them. That’s not to have a go at clients but anyone familiar with the concept of scope creep will understand this. The key here is to get an idea of what’s in and what’s out. I recommend the use of a technique defined many years ago and states simply what’s in, what’s out and what might be considered. It’s a great tool for getting clients to understand all the elements involved in a project.
To be resolved
Once again, an important consideration. Many sites are constructed without a proper regard of who it is aimed at. Saying that it is aimed at the client’s customers is not enough. We need to understand the demographics of that customer base and if we have to target one part of that customer base, which one is it. In reality most sites have a number of target markets that need to be considered. What proves to be a more practical approach is to think of it in terms of primary, secondary and tertiary target markets.
The main difference between traditional software projects and web development projects is the nature of content. In most web projects, there is a large amount of content. A web site can be both an application or a hypertext system (eg. pages of content). Most projects are a combination of the two and both aspects must be address. For projects with a large amount of content, information architecture and design become a very important element of the project’s success.
In it’s most simple form, information architecture can be thought of as a sitemap. It’s the logical structuring of content to suit the purpose of the site. Although this seems like a straight forward task, the importance of this should not be underestimated. A well structured site will be easier for people to use and will be much easier to maintain. A good example of a poorly architectured site is one in which users quickly turn to the search facility to find what they are after.
Most people now understand the concept of a site map and many can put a reasonable one together. The next level, which is far more sophisticated is to then look at the design of information on the page, ie the information design. The current trend is towards the three column layout with a header and footer.
Site ID / Logo
Miscellanoues Links, eg. Terms & Conditions, Sitemap, Contact Us…etc
This is where features come into play. This is where the “application” aspect of the website is defined. The key here is to break the project down into chunks of work that can be captured in the client’s language and that each “chunk” is between 2 hours and 2 weeks of work
An integral part of FDD is the weekly report which I have adapted for smaller projects. In a larger FDD project, the report would look into the details of how many features had been started, not started, in progress, finished and late. From this an accurate percentage complete figure can be derived. In small projects, it’s not always necessary to go to this level of detail. However, it is important to track progress and report this to the client regularly. I recommend doing this in two ways as set out below.
A daily meeting with the team. This is a common element amongst agile processes, in Scrum it’s called the Daily Scrum or daily stand up meeting. It is a short meeting in which each person states what they did the previous day and what they plan to do today. There are a number of goals for this meeting.
make sure no-one is falling behind
make sure everyone knows what the rest of the team is working on
ensure that any barriers or problems are raised quickly
help foster collaboration within the team.
From a project management perspective it’s an extremely effective tool to keep developers on track. When people promise to do something and they know they will be asked the next day in front of others whether it has been done, it certainly makes them think twice about what they promise and not following through! Also, it has a strong social aspect which makes a huge difference to team work.
The key here is to provide the client with updates on progress on a weekly basis. There are many reasons for this, the first of which is providing a discipline for the project manager to review the project on a high level. The progress report doesn’t need to be complex or comprehensive, it simply needs to state what has been done and if there are any issues that need to be resolved. I use the following template which has proved to be very effective
I simply stated what has been completed in the previous week. Helps to give a sense of progress and satisfaction.
This is a valuable technique in getting things done. Often the dependency is something the client has to deliver, by putting it down in writing, it makes it clear who is responsible for delays.
I can’t stress enough how important it is to clearly state any assumptions, when not stated openly, they can often cause major issues. Eg. a developer may assume that data provided will be in a tabular, delimited format that is easy to import, the client however delivers it as a word file or worse, in a graphic format such as quark or pagemaker.
Every project has issues, these need to be captured and written down.
Hopefully, as the project progresses, the issues raised are resolved. It’s good form to capture that so that should questions be asked down the track, you can always go back to the resolution from that progress report. Helps with clients that have a habit of changing their minds.
This is a large part of all FDD projects and in those projects is called the KMS (knowledge management system). All information for the project, all documentation, meeting notes, progress reports…etc should be captured in website that both the project team and client can access at anytime. It helps to ensure consistency and ensure a central repository.
There is still no silver bullet, adapting FDD for web development goes a long way to dealing with many of the issues that arise in web development but it is not the complete answer. Areas such as requirements gathering, visual design, testing & deployment aren’t covered even though they are needed for every project. But given the lack of web methodologies out there, having something that is effective, albeit incomplete, it is a big step forward.