PROJECTS change. This simple fact is not fundamentally due to a lack of planning or incompetence on the part of project managers and project developers. Rather, change is an inherent characteristic of any growing entity. Embedded projects grow as much as they are built. Living things adapt to their environment. The environment surrounding any embedded project is ever in flux. Budgets change. Resources change. Schedules change. Competition changes. Customer needs change. Even if this changing environment could be eliminated, another form of change would continue to affect embedded projects. A project learns as it grows and must change in response to this learning. That is, as features come to fruition, the developers, users, customers, and managers become more fully aware of the project's reality. First, by "traditional project management" we refer mainly to any methodology where project development is viewed as a specialized version of manufacturing or as a construction project. This type of project management is identified by its sequential phases of design, implementation, and testing (the "waterfall" approach) planned out through critical path analysis (usually represented via Gantt charts). Second, we address here only those projects that include any sort of variability or unknowns in their requirements
Agile methodologies such as eXtreme Programming (XP), SCRUM and Feature-Driven Development strive to reduce the cost of change throughout the project development process. For example, XP uses rapid iterative planning and development cycles in order to force trade-offs and deliver the highest value features as early as possible. In addition, the constant, systemic testing that is part of XP ensures high quality via early defect detection and resolution. In spite of some early success with agile methodologies, a number of factors are preventing their widespread adoption. Agile methodology advocates often find it difficult to obtain management support for implementing what seem like dramatic changes in application development. These methodologies require developers, managers and users alike to change the way they work and think. For example, the XP practices of pair programming, test-first design, continuous integration, and an on-site customer can seem like daunting changes to implement. Furthermore, these methodologies tend to be developer-centric and seem to dismiss the role of management in ensuring success. Traditional management theory assumes that:
- Rigid procedures are needed to regulate change
- Hierarchical organizational structures are means of establishing order
- Increased control results in increased order Organizations must be rigid, static hierarchies
- Employees are interchangeable "parts" in the organizational "machine"
- Problems are solved primarily through reductionist task breakdown and allocation
- Projects and risks are adequately predictable to be managed through complex up-front planning
Within this context, it is small wonder that the new methodologies appear informal to the point of being chaotic, egalitarian to the point of actively fostering insubordination, and directionless in their approach to problem solving. We believe that the slow adoption of agile methodologies stems mainly from this misalignment between the fundamental assumptions of traditional management and those of the new agile development methodologies. As such, there is a significant need for a change in assumptions and a new management framework when working with agile methodologies.
Specifically, we have begun to build the notion of complex adaptive systems (CAS) into our management assumptions and practices. Complexity scientists have studied the collective behaviour of living systems in nature such as the flocking of birds, schooling of fish, marching of ants and the swarming of bees. They have discovered that, while the individual "agents" in these complex adaptive systems possess only local strategic rules and capacity, their collective behaviour is characterized by an overlaying order, self-organization, and a collective intelligence that is greater than the sum of the parts. The theory of CAS has been applied successfully in several areas - economics, life sciences and more recently, to management. These concepts of CAS led to the inspiration that like the XP team, project managers also need a set of simple guiding practices that provide a framework within which to manage, rather than a set of rigid instructions. Following these practices, the manager becomes an adaptive leader - setting the direction, establishing the simple, generative rules of the system, and encouraging constant feedback, adaptation, and collaboration. This management framework, covered in detail in Section 4, provides teams implementing agile methodologies with:
- An intrinsic ability to deal with change
- A view of organizations as fluid, adaptive systems composed of intelligent living beings
- A recognition of the limits of external control in establishing order, and of the role of intelligent control that employs self-organization as a means of establishing order
- An overall problem solving approach that is humanistic in that:
- It regards employees as skilled and valuable stakeholders in the management of a team.
- It relies on the collective ability of autonomous teams as the basic problem solving mechanism.
- It limits up-front planning to a minimum based on an assumption of unpredictability, and instead, lays stress on adaptability to changing conditions.
The Problem: Project Management as Uninspired Taskmaster
Traditional project lifecycle development methodologies grew out of a need to control ever-larger development projects, and the difficulties of estimating and managing these efforts to reliably deliver results. These methodologies drew heavily on the principles from engineering such as construction management. As a result, they stressed predictability (one has to plan every last detail of a bridge or building before it is built), and linear development cycles - requirements led to analysis which led to design which in turn led to development. Along with predictability, they inherited a deterministic, reductionist approach that relied on task breakdown, and was predicated on stability - stable requirements, analysis and stable design. While these methodologies may have worked for some organizations in the past and may still work in some circumstances, for many companies these methodologies only added cost and complexity while providing a false sense of security that management was "doing something" by exhaustively planning, measuring, and controlling. Huge costs were sunk in premature planning, without the rapid iterative development and continuous feedback from customers that we have come to realize are prerequisites for success today. The results are stark - repeated, public failures such as the London Ambulance System and the Denver Airport Baggage system earned the project industry a reputation for being "troublesome" with huge cost overruns and schedule slippages.
Consider the results of the Standish Group's CHAOS surveys. In the first survey, it was estimated that only 18 percent of all project projects were considered successful, 31 percent were failures and 53 percent were challenged. Comparatively, the 1998 figures showed a marked improvement in which 26 percent were successful, 46 percent were challenged and 28 percent were failures. The study attributed the increase in success to scaling the size of projects back to manageable levels using smaller teams. This result is clearly in line with the principles of agile methodologies. Furthermore, many established project management practices still apply to agile development projects - with some adaptation and a strong dose of leadership. While managers designed traditional methodologies in an effort to control projects, the technical community gave birth to agile methodologies in response to their frustrations with traditional management (or lack thereof) and the resulting impact on their products and morale. For example, the principles of XP are focused almost entirely on the development process. While the technical community has championed these principles, very little has been written about the management side of agile development projects. The traditional project manager is often seen as a "taskmaster" who develops and controls the master plan that documents (often in excruciating detail) the tasks, dependencies, and resources required to deliver the end product. The project manager then monitors the status of tasks and adjusts the plan as necessary. So for many managers comfortable with traditional methodologies, the prospect of implementing agile methodologies on their development projects can be daunting. But it doesn't need to be. In fact, independent of agile methodologies, other trends in project management indicate a point to a convergence between the management community and the technical community.
The Solution: Project Manager as Visionary Leader
The best project managers aren't just organizers - they combine business vision, communication skills, soft management skills and technical savvy with the ability to plan, coordinate, and execute. In essence, they are not just managers - they are leaders. While this has always been the case, agile project management places a higher importance on the leadership skills than ever before. For example, XP teams create and monitor their own iteration plans in collaboration with the customers. The customer creates stories (features) and prioritizes them based on business value. Agile methodologies free the project manager from the drudgery of being a taskmaster thereby enabling the project manager to focus on being a leader - someone who keeps the spotlight on the vision, who inspires the team, who promotes teamwork and collaboration, who champions the project and removes obstacles to progress. Rather than being an operational controller, the project manager can become an adaptive leader - if she can relinquish her reliance on old style management. The basic phases of an agile development project are really no different from those of any other project. He still must define and initiate the project, plan for the project, execute the plan, and monitor and control the results. But, the manner in which these steps are accomplished is different and require the project manager to retrofit what they know about traditional management to a new way of thinking - the thinking of complex adaptive systems. The practices outlined below provide a framework for project managers working in this new world.
The Means: An Agile Project Management Framework
The authors have applied XP successfully on several projects over the past years, and evolved the use of XP practices as an integral part of a CAS inspired framework for agile project management, as described in Section 4.2. Section 4.1 provides a guiding philosophy of the team as a complex adaptive system.
A CAS-Based Project Management Framework: Six Practices for Managing Agile Development Project
We have established a CAS-based project management framework with six Agile Project Management (PM) practices for managing agile development projects - Guiding Vision, Teamwork and Collaboration, Simple Rules, Open Information, Light Touch and Agile Vigilance. Together these practices help us to manage our teams as complex adaptive systems while allowing us the freedom to overlay our own personal leadership styles. The six practices build on the fundamentals of CAS, as shown in Table 1. These practices are explained in further detail in Sections 4.2.1 through 4.2.6.
Practice #1: Guiding Vision - Establish a guiding vision for the project and continuously reinforce it through words and actions.
As articulated by Margaret Wheatley , when a project vision is translated into a statement of the greater purpose and dreams of the organization, and communicated to all members of the team, it serves as a field that has a powerful effect on their behaviour. It can permeate the project environment and influence team behaviour in extremely positive ways, much more so than a simple task can. A real example of this principle is the use of the "commander's intent" in the U.S. Army. The Army knows that its leaders cannot be everywhere in the field of combat controlling all the decisions. Therefore, Army leaders clearly establish the "commander's intent" to serve as a guide on which soldiers can base their own initiatives, actions and decisions. Thus, even if the mission falls on the shoulders of the lowest ranking person, she must be able to understand and carry out the mission. Likewise, the agile manager, can guide the team and continuously influence team behaviour by defining, disseminating and sustaining a guiding vision. At the outset of the project, work closely with the customer to understand the vision for the project, how it is expected to support business goals, and how it will be used. A strong grasp of the vision will help the team through difficult decisions about business value and priority and keep them focused on and inspired by the ultimate goal.
Practice #2: Teamwork & Collaboration - Facilitate collaboration and teamwork through relationships and community.
The project manager's role is to actively facilitate collaboration and establish the conditions for good relationships. Good relationships among team members start with the project manager's relationship with the team members. Know what makes each of them tick outside of work and what motivates each of them at work. He should help team members get to know each other by creating opportunities and the right conditions. Opportunities can be created from planning games, everyday interaction, and special events. To set the right conditions, he must establish an environment in which team members treat each other with respect. He may even need to intervene to stop disrespectful behaviour. Some people may not be comfortable bringing their technical problems to the group.
The project manager must monitor the team dynamics and decide when to intervene. As the project progresses, continue to look for special opportunities to get to know people better and to help the team know each other. For example:
- Establish a regular day for group order-in or potluck lunches
- Giving team members fun (positive!) nicknames
- Celebrating successes and milestones with nominal gifts that reflect knowledge of staff interests (e.g., music, gift certificates, special foods).The team that laughs and plays together works together better.
Practice #3: Simple Rules - Establish and support the team's set of guiding practices.
In a CAS, agents follow simple rules, but their interactions result in complex behavior emerging from the bottom-up over time. For example, birds in a flock follow simple rules such as avoiding objects, keeping pace and staying close to other birds. By following these simple rules, flocks of birds exhibit complex, collective behavior by flying in formation for long distances and adapting to changing conditions along the way. Similarly,These XP practices provide the team with a flexible structure within which to work Take a leading role in encouraging the team to try certain practices about which team members may be doubtful. In applying the XP practices, he must set up simple generative rules that are just enough to provide clear boundaries, but not so much as to restrict the autonomy and creativity of the team. Throughout the project, appropriately point out when practices are not being followed and seek to understand why, looking for opportunities to adjust and improve on the practices or their practical use.
Practice #4: Open Information - Provide open access to information.
For an agile team to be able to adapt, information must be open and free flowing. Traditional managers have long prevented this openness and freedom because of a fear that it will result in chaos. Because of this fear, traditional managers have controlled information and meted it out on a "need to know" basis. On traditionally managed projects, teams often feel like they don't know what is going on - only the project manager has the "master plan" and only the project manager interacts with project sponsor. In the agile world, information is freed to leverage its power. To promote open information, he can try a variety of techniques:
- Place team members within close proximity of each other whenever possible.
- Make use of information radiators such as whiteboards, charts, etc to disseminate information.
- Establish daily status meetings to promote the flow and exchange of information.
Practice #5: Light Touch - Apply just enough control to foster emergent order.
We believe that control and order are related in a way. Without any control at all, there exists a certain level of order due to self-organization, depending on the team skills and dynamics. Initially, as control increases, order increases somewhat linearly, and reaches a narrow plateau quickly, decreasing very rapidly afterwards. Of course, the conventional view holds that the initial condition of no control starts off without any order atall, with an increasing linear relationship. Visionary control is a delicate mix of emergent and imposed order. To impose order, he must impose some control, but do it with a "light touch". With a progressive "light-touch" mindset, lay out project plans at a high-enough level to give the team room for innovation, creativity and rapid response to dynamic environments. Ensure that the project plans are synchronized with her guiding vision, and that they are based on functionality to be delivered and not tasks
Practice #6: Agile Vigilance - Constantly monitor and adjust.
In leading a team by establishing a guiding vision, fostering teamwork and cooperation, setting simple rules, championing open information, and managing with a light touch, the job of the agile manager has been likened to herding cats - each person has his or her own ideas, and is likely to behave in accordance with those ideas. The agile manager, therefore must be continually vigilant to merit the mantle of leadership: monitoring progress, and keeping a finger on the pulse of the development team.
- Reinforce the guiding vision at every opportunity - examine project decisions to see whether they line up with the vision.
- Continually encourage teamwork and collaboration. Establish simple rules, but take every opportunity to conduct process reflections: regularly examine what works and what needs improvement
- Operate with a light touch. Intervene quickly, but wisely to solve personnel issues. Motivate and reward initiative, but manage expectations. Recognize and encourage self-organization, but disallow cliques.
The lack of guidance for project managers of agile development projects has been a gaping hole in the project development community over the past several years. The contrast between the world of agile project development and traditional project management has left many managers wondering what their role should be. By viewing the agile development team as a complex adaptive system and the manager as an integral part of that system, we have begun to develop a framework for managers. This framework of practices is meant to overlay the practices of existing agile methodologies such as XP, and provide clear guidelines for the visionary leadership of projects that use them. These six practices of agile project management do not provide a sure-fire recipe for success.