Agile is Fragile

Handle with Care. Agile is very easy to Break, very hard to Nurture and Preserve.

Six years since Agile was born, when I revisit the original Agile Vision and Values, it feels nostalgic. The Agile Paradigms that literally caused a revolution in the way people build software today seems to be begging its communities to realign.

Many in the Agile Community today, have lost sight of the original Vision of Agile and have got caught in deep theology conflicts as we see it in emerging practices and trends in Agile, loosing its “focus and alignment” to the basic Values and Principles enshrined in Agile.

I keep asking myself “Is Agile today really Agile? “ Join me in evaluating what people are doing, when they claim that they are practicing Agile.

Here are some real-life examples of Agile Methodology Practices implemented as Agile, which are completely out-of-sync with the original Vision and Values of Agile:

1. Agile Software Development is done through Fixed Bid Contracts. (some Agile Methodologies like Scrum are coaching organizations how to build such contracts for Agile Projects)
2. Agile Software Development is done on a set of defined process framework. (some Agile Methodologies like Scrum have been very prescriptive and lay down rigid practices as rules of the game)
3. Agile Methodologies are offering Certification Programs for individual and organizations for conformance to the processes and practices that they define.
4. Customers are not educated on the Agile Values and therefore they do not actively collaborate and participate along with the team in their software development endeavors. Agile teams to overcome this create a Proxy Product Owner who finally is unable to fill in for this purpose and the project fails.
5. While Agile Values in principle can be nurtured only in collocated small teams, organizations scale up Agile to distributed not collocated teams, without due regard for its implications and finally blame Agile for their failures.
6. Management of Software Development Organizations do not change their traditional polices and are unable to nurture the Agile Team Values due to which the agile teams become inefficient and unproductive
7. Managers of Agile Software Projects do not understand the principles of Agile Project Management and try to manage Agile Teams with their traditional ways of formal command and control style of management.
8. Management does not provide the open-space work environment that the Agile Teams need for nurturing face to face informal communications and information radiators are missing.
9. Management does not take enough care when composing cross-functional Agile teams, due to which the teams lack the spirit of integration and team work and are not able to implement the Agile Engineering Practices well due to inherent skills lacunae.
10. While many Agile Teams are serious about Agile, the Manager, the Management and often Customers too, do not keep up their commitment for collaboration, trust, empowerment and support to the team, and become the reason for teams’ failure to deliver value to them.
11. I have seen software organizations implement Agile in a CMMi style and expect the Agile Teams to create their Agile Practices within the framework of the CMMI process.
12. Agile Methodologies like Scrum is inculcating a spirit of conflicting stance into Agile Teams teaching them skills to guard themselves against the Management and the Customers. This totally goes against the imperative Value of Collaboration, Transparency and Trust in Agile. (the Pig and the Chicken story is a misnomer paradigm reinforced by the Scrum Master being labeled as a person who protects the team from outside adversaries like the Customers and the Management)

What pains me is the way our community is innovating new principles and practices in Agile that are simply not aligned to the original Agile Vision and Values. They seem to make their “Branded Practices” more important than the Principles and Values that drive Agile. In the processes such branded practices are becoming rote, as they seem to have forgotten to align each of their new found practices to the original principles and values in Agile.

For those who are new to Agile, I would like to recall what Agile was meant to be.

Agile Vision

To create a new alternative set of paradigms that can help people build good software and deliver true value to customers, in the face of volatile scope and unpredictable development environments

Agile Vision Rationale

The software industry was going through a crisis as many software development projects were failing to deliver “Value to Customers” (refer Standing Survey Reports), in the face of fast changing technology environments, volatile requirements scenarios, ambiguities in our ability to predict people capabilities and the need to build better software, faster and cheaper, to make best use of business opportunities that had really small and shrinking time windows.

Agile Values

The Agile Values were a set of paradigms that could reinforce the deficiencies in the then methods for software development and still enable them work in such situations.

They were created as a set of strategies to beat the problems of uncertainties and volatilities in software development environments

1. Customer Collaboration (over contract negotiations) : By actively collaborating with the customers in jointly developing software together, the inherent weaknesses in the development system could be curtailed to a large extent. The weaknesses could include those that are caused through formal documentation (translation errors) and due to poor communication and interactions with the customers in eliciting their true needs, clearly, completely, correctly and consistently. It was also known that the developers would lack the business domain competence and the customers would be ignorant of technical implications of technology choices and therefore, when both would work collaboratively and jointly, these weaknesses could be overcome. Such collaboration with customers, necessitate changes in the way development organizations decide their business policies and customer management practices for providing their services to customer organizations. So contracting with customers on fixed terms was not possible in such situations.
2. People Interactions (over processes and tools) : Software Development needs extremely high levels of creativity, innovation, commitment and discipline. Knowledge workers cannot bloom in a strictly defined process environment. They need a degree of interactions, informal face to face communication with stakeholders, ability to change directions midway, learn through the way and build the best possible solution in a given context. For such activity to happen, teams have to be cross-functional, empowered and self organized through empirical process and tacit collective wisdom. They work as a team and compliment one others skills and competencies, learn faster and contribute better in an environment of high team morale and individual motivation. To succeed, they need to nurture team values of trust, transparency, commitment, honesty, respect, courage, discipline and proactive and informal communication both ways. This enables teams produce highest value to customers. The premise is that subjecting knowledge workers to a set of defined processes and controlling them over a set of assigned tasks through automated monitoring and tracking tools, does not produce good results. It was therefore imperative that Management of Development Organizations have to make serious paradigm shifts and create revised organizational structure and management policies to manage their people, processes and projects to stay consistent with Agile Values.
3. Working Software (over comprehensive documentation). The traditional contemporary software development and project management practices were built over predictive planning and structured management control systems. It was based on process maturity models and believed in the verification approach to quality ( do things the right way, right things well get automatically built). In an unpredictable situation, this would not work. Agile Values brought about a Validation Approach to delivering Quality, and Quality was defined as Customer Value. Agile innovated new paradigms for a Validation Approach to software development through empirical process that adaptively evolves through contextual feedback and learning. ( find the right ways to build the right thing and keep iteratively validating if right things are being built)
4. Responding to Change (vs following a plan) To compliment the above Values and implement true Agility, the Agile Community embraced Incremental and Iterative software development paradigms in an informal context to help them learn and validate faster and allow for adaptive and iterative planning to meet the Project Goals more efficiently. This was to replace the sequential predictive ways of software development (like Waterfall Model, V Model, Incremental Model) which did not have the ability to keep the cost of change curve flat. However this way of developing software iteratively and incrementally, needed software programmers to learn new software engineering skills that includes the “how-to” of Incremental Architecture, Refactoring, Evolving Design, Test Driven Development, Continuous Integration, Test Automation, Test Refactoring, Coding Standards, without which they will not be able to build working software consistently.

I wonder what we could do to bring back Agile Methodologies on the right track all over Again ?

Scrum is not Agile

More Customers, More Software Development Organizations, More Software Professionals, more than ever before, are today waking up to the "Agile" that was started 6 years ago and called the Agile Alliance . In the haste to embrace Agile, many are suffering in transitioning to a new set of paradigms without first understanding its implications.

The understanding of Agile, sometimes, I feel is so hollow, that some people have started equating their methodologies and tools to Agile. Once such case in point is Scrum. With aggressive marketing and high sales pitch, Scrum, one of the methodologies under the Agile Umbrella has already created 12000 Certified Scrum Masters and plan to reach 20,000 of them by end of this year. So much it has been marketed worldwide, that, today, the perception is that Agile is Scrum and Scrum is Agile.

Organizations are picking up Scrum overnight and scaling it up to Multi-location, Multi-project, Distributed Software Development Projects, without giving due attention to the rationale of its implementation and suffering the consequences.

I named this post "Scrum is not Agile" not to say that Scrum is not Agile. but to say "Agile is not Scrum" loudly. Give me some of your time to understand why I say so. I welcome your feedback and comments.

We all talk about Agile. What is Agile ?

Agile is a "School of Thought" that is best described in the Agile Manifesto. It is a set of Values that provide ways to build good software, faster, cheaper and better, especially, in the face of volatile requirements and unpredictable development environments.

Agile Software Development Paradigms attempt to counter the volatility and uncertainties through iterative cycles of "Speculation-Collaboration-Learning" that enables the participants to minimize the adverse effect of risks and maximize delivery of Value to the Customers.

These Paradigms are built over a set of complex assumptions, the understanding of which is a key imperative , if one wishes to reap the true benefits of Agile.

I have tried to analyze these imperatives to enable those implementing Agile, understand, "how-to" harvest and realize such benefits, efficiently, in their respective software development endeavours.

Where do we start ? Let's start by understanding how Agile promises to deal with the following issues that it targets :

1. Volatile Scope (Customer Requirements can keep changing or evolving over time and we cannot know much of it when we start, yet we should be able to keep the "cost of change curve" , flat.
2. Unpredictive Development Environment ( most of the times, we develop software in scenarios where much of what we need to know, cannot be known upfront. Such uncertainties could range from issues related to technology, team capability, cost/time estimation or even our ability to upfront design a good solution that fully addresses the real needs of our customers)
3. Create higher Value to Customers ( building the right software faster, cheaper and better, with a motivated set of team players)

The "Agile Strategy" employed to achieve this is based on the following four core principles:

1. Customer Collaboration : change the way we work with the customers. Collaborate with them to minimize risks and unpredictabilities. We cannot achieve this through any other means. Agile can work only if Customers and Developers, actively collaborate and work together as one team. To do this effectively, both customer and supplier organizations may have to undergo a serious paradigm shift in their Business Policies.
2. Motivated Team : we can improve the productivity of knowledge workers only through improving their Motivation. We can harness the power of creativity and innovation only through empowered self-organizing, cross-functional teams working in an environment that fosters transparency, mutual trust, informal face-to-face communication, team-play and accelerated learning. To nurture such teams, management policies and managers may need major paradigm shifts in the way we manage people resources today.
   
3. Iterative Software Development: we can produce high value to customers, in an environment of high volatility and uncertainties, only through iterative software development that helps us produce the right software, right through active and ongoing collaboration and feedback from our customers. This is the PDCA approach for continuous improvement in the way we do, what we do. To be able to implement iterative software development, teams may need new software engineering skills that include incremental architecture, refactoring, test driven development, continuous integration, test automation. The engineering managers may also have to undergo a torturous paradigm shift in the way they manage software quality assurance and software quality control.
   
4. Agile Project Management: To achieve this, the way we manage our teams and projects have to change. Agile Project Management introduces the new paradigms of empowerment, servant management and collaborative leadership. It aims at providing the right work environment, the right resources, the right support and the right guidance for the Agile Teams to succeed. For this to work, management of organizations and their managers will have to undergo a total cultural and value re-orientation of how they should be managing projects, people and processes.
   

Now all the four items listed above are critical imperatives for Agile to work. In implementing Agile, we have to ensure that all these four critical success factors are evenly aligned and each is effective, failing which the velocity of the Value streaming can suffer seriously. This is best illustrated by the figure below.


The Value is streamed to the Customers by the Developers through a pipe that has four sections. Each section may be constrained by the effectiveness of that factor in play. Therefore going by the Goldratt's Theory of Constraints, the rate of throughput (which the value stream here), is affected directly by the bottleneck in the system. Bottleneck is the core constraint of the system. So even if one part of this system is not effectively managed, notwithstanding how well the other three parts of the system are managed, we will not be able to improve the rate at which Value is delivered to the Customer.

This is the SECRET that we all need to know and understand before implementing Agile, if we were to HARVEST the true benefits of AGILE.

Coming back to Scrum.

Scrum is only an Agile Project Management Methodology under the Agile Umbrella and addresses only one aspect of Agile Software Development Paradigms out of the four Agile Imperatives described above. It does not address, as it is today, any other critical success factor of Agile Software Development.

So, Scrum Implementers, especially those who do in haste, enlarge only one section of the pipe and are therefore not able to improve the rate of Value Delivery to the Customers, which can only happen when all the four sections of the pipe are declogged simultaneously.

Scrum therefore fails, more often than it succeeds, if it is implemented as an Agile Software Development Methodology in a software development environment. It can only help manage one part of our problems. It is no way a PANACEA.

Thanks for your time. I know we can do better. I sincerely look forward to your feedback and comments. I will be eager to know if this helps in improving the way we can use Agile Paradigms, Agile Methodologies and Agile Tools on our future Agile Projects.