Monday, March 19, 2007

Scrum is not Agile

More Customers, More Software Development Organizations, More Software Professionals, more than ever before, are today waking up to the "Agile" that was started 6 years ago and called the Agile Alliance . In the haste to embrace Agile, many are suffering in transitioning to a new set of paradigms without first understanding its implications.

The understanding of Agile, sometimes, I feel is so hollow, that some people have started equating their methodologies and tools to Agile. Once such case in point is Scrum. With aggressive marketing and high sales pitch, Scrum, one of the methodologies under the Agile Umbrella has already created 12000 Certified Scrum Masters and plan to reach 20,000 of them by end of this year. So much it has been marketed worldwide, that, today, the perception is that Agile is Scrum and Scrum is Agile.

Organizations are picking up Scrum overnight and scaling it up to Multi-location, Multi-project, Distributed Software Development Projects, without giving due attention to the rationale of its implementation and suffering the consequences.

I named this post "Scrum is not Agile" not to say that Scrum is not Agile. but to say "Agile is not Scrum" loudly. Give me some of your time to understand why I
sa
y so. I welcome your feedback and comments.

We all talk about Agile. What is Agile ?

Agile is a "School of Thought" that is best described in the Agile Manifesto. It is a set of Values that provide ways to build good software, faster, cheaper and better, especially, in the face of volatile requirements and unpredictable development environments.

Agile Software Development Paradigms attempt to counter the volatility and uncertainties through iterative cycles of "Speculation-Collaboration-Learning" that enables the participants to minimize the adverse effect of risks and maximize delivery of Value to the Customers.

These Paradigms are built over a set of complex assumptions, the understanding of which is a key imperative , if one wishes to reap the true benefits of Agile.

I have tried to analyze these imperatives to enable those implementing Agile,
understand, "how-to" harvest and realize such benefits, efficiently, in their respectiv
e software development endeavours.

Where do we start ? Let's start by understanding how Agile promises to deal with the following issues that it targets :

  1. Volatile Scope (Customer Requirements can keep changing or evolving over time and we cannot know much of it when we start, yet we should be able to keep the "cost of change curve" , flat.
  2. Unpredictive Development Environment ( most of the times, we develop software in scenarios where much of what we need to know, cannot be known upfront. Such uncertainties could range from issues related to technology, team capability, cost/time estimation or even our ability to upfront design a good solution that fully addresses the real needs of our customers)
  3. Create higher Value to Customers ( building the right software faster, cheaper and better, with a motivated set of team players)
The "Agile Strategy" employed to achieve this is based on the following four core principles:
  1. Customer Collaboration : change the way we work with the customers. Collaborate with them to minimize risks and unpredictabilities. We cannot achieve this through any other means. Agile can work only if Customers and Developers, actively collaborate and work together as one team. To do this effectively, both customer and supplier organizations may have to undergo a serious paradigm shift in their Business Policies.
  2. Motivated Team : we can improve the productivity of knowledge workers only through improving their Motivation. We can harness the power of creativity and innovation only through empowered self-organizing, cross-functional teams working in an environment that fosters transparency, mutual trust, informal face-to-face communication, team-play and accelerated learning. To nurture such teams, management policies and managers may need major paradigm shifts in the way we manage people resources today.
  3. Iterative Software Development: we can produce high value to customers, in an environment of high volatility and uncertainties, only through iterative software development that helps us produce the right software, right through active and ongoing collaboration and feedback from our customers. This is the PDCA approach for continuous improvement in the way we do, what we do. To be able to implement iterative software development, teams may need new software engineering skills that include incremental architecture, refactoring, test driven development, continuous integration, test automation. The engineering managers may also have to undergo a torturous paradigm shift in the way they manage software quality assurance and software quality control.
  4. Agile Project Management: To achieve this, the way we manage our teams and projects have to change. Agile Project Management introduces the new paradigms of empowerment, servant management and collaborative leadership. It aims at providing the right work environment, the right resources, the right support and the right guidance for the Agile Teams to succeed. For this to work, management of organizations and their managers will have to undergo a total cultural and value re-orientation of how they should be managing projects, people and processes.
Now all the four items listed above are critical imperatives for Agile to work. In implementing Agile, we have to ensure that all these four critical success factors are evenly aligned and each is effective, failing which the velocity of the Value streaming can suffer seriously. This is best illustrated by the figure below.


The Value is streamed to the Customers by the Developers through a pipe that has four sections. Each section may be constrained by the effectiveness of that factor in play. Therefore going by the Goldratt's Theory of Constraints, the rate of throughput (which the value stream here), is affected directly by the bottleneck in the system. Bottleneck is the core constraint of the system. So even if one part of this system is not effectively managed, notwithstanding how well the other three parts of the system are managed, we will not be able to improve the rate at which Value is delivered to the Customer.

This is the SECRET that we all need to know and understand before implementing Agile, if we were to HARVEST the true benefits of AGILE.

Coming back to Scrum.

Scrum is only an Agile Project Management Methodology under the Agile Umbrella and addresses only one aspect of Agile Software Development Paradigms out of the four Agile Imperatives described above. It does not address, as it is today, any other critical success factor of Agile Software Development.

So, Scrum
Implementers, especially those who do in haste, enlarge only one section of the pipe and are therefore not able to improve the rate of Value Delivery to the Customers, which can only happen when all the four sections of the pipe are declogged simultaneously.

Scrum therefore fails, more often than it succeeds, if it is implemented as an Agile Software Development Methodology in a software development environment. It can only help manage one part of our problems. It is no way a PANACEA.

Thanks for your time. I know we can do better. I sincerely look forward to your feedback and comments. I will be eager to know if this helps in improving the way we can use Agile Paradigms, Agile Methodologies and Agile Tools on our future Agile Projects.

14 comments:

Sachin Palewar said...

You seem to be right to me, that's why we find many organisations using scrum for project management along with XP for development and I have heard it makes a great combo. However since you brought up this interesting thought and gave a wonderful analogy of Goldratt's TOC, I would be interested to know if you have any suggstions for remaining 2 sctions of the pipe, if we think that other 2 are handled well by XP and Scrum combined?

Stefan Roock said...

Just to be explicit: What is the second section addressed by XP: Iterative Development / Incremental Design?

Siddhi said...

I'm a bit confused about what you call agile project management. Providing right resources and guidance is just good management practice, Agile or otherwise.

What I understand as APM is what you have put as the separate sections of the pipe - iterative and collaborative.

Scrum by itself is iterative, and emphasizes customer collaboration (via the product owner role). So doesn't you think Scrum includes the whole pipe?

Jeff Langr said...

I think Scr*m proponents do indicate that it is not a complete solution, and is best combined with solid engineering practices (such as those that come from XP--which begs the question, why not just do XP?).

Still, I suspect far too many shops are implementing Scr*m without a concern for the long-term effect caused by rapid iterative development. The lack of heavy disclaimers for Scr*m in this respect is close to malpractice, as far as I'm concerned.

Oh yeah--don't let 'em intimidate you, please keep posting.

Anonymous said...

Jeff I is on to something. What are the long-term effects of rapid iterative development? If you're doing XP, there's built-in correction (refactoring on the micro level and release planning on the macro level) but I have yet to see any sort of serious discussion of "design for maintainability" anywhere - let alone in the agile community.

But back to the topic. Recognizing that Scrum is only part of the equation is crucial and I thank you for that.

S M Kripanidhi said...

I sincerely thank all of you for your feedback and comments. Please keep it going.

We can all learn from many different schools of thoughts in management to help us innovate the way we manage our people and customers.

In Agile, the Second Edition of Extreme Programming by Kent Beck is a great book on Agile Values, Principles and Practices. Industrial XP (IXP) is also great set of concepts for practical implementation of these Agile Values in an incremental and effective manner.

One should also learn from the Service Quality Management paradigms like SERVQUAL and the way service organizations nuture and harvest the best of self organizing teams, collaborative leadership and supportive management.

Theory of Constraints and Critical Chain is another great set of ideas on systems approach to continuous improvements. Lean Principles and Just in Time are also very useful in real time.

We should always make sure we know, why we do, what we do, the way we do it.

Cheers

Anix said...

Kripanidhi

Thanks for putting your thoughts on Forum but I am sorry and I am totally disagreeing with your thoughts.

As Jim HighSmith mentioned,

1)Agility is the ability to create and responds to change.

2)Agility is the ability to balance flexibility and structure.

Scrum supports the above mentioned very important points very well.

As you have mentioned on your blog, “Scrum is only an Agile Project Management Methodology under the Agile Umbrella”. I would say SCRUM is a common sense; SCRUM is not a solution, SCRUM itself is not a problem but SCRUM is a mechanism to make your problems more visible. SCRUM is a more discipline and less process which agile supports.

SCRUM supports all the important factors of “Agile”, and if you use SCRUM in really proper way, you will get only success.

I am not against your thinking but if SCRUM doesn’t work for you in terms of Agility, it doesn’t mean that SCRUM is not agile.

Your thoughts may confuse other people. Ken Schwaber has already given his nice feedback on Scrumdevelopment yahoo group.


Regards,

Aniket
Certified Scrum Master

S M Kripanidhi said...

Thanks for your feedback. Ken Schwaber does not like the idea of discussion on the merits or demerits of the Certified Scrum Master program in his scrumdevelopment@yahoogroups.com forum. We have therefore been discussing it on the IXP forum (industrialxp@yahoogroups.com)for over 3 weeks now. Ken also participates there and provides his point of view. If you are interested, all of you are welcome to view the several threads on this at the IXP forum.

Unknown said...

Its a nice article, but you have done a interesting comparison of agile and TOC. Can you share more details about TOC relation to agile practices, I am bit confused about the relationship you have provided.

S M Kripanidhi said...

TOC and Agile.

Agile is based on Empirical Process Evolution Paradigms where teams with their Collective Tacit Knowledge and through Structured Retrospectives innovate new practices that work and eliminate waste systematically.

TOC is a systems approach to continuous improvement and using the concepts of TOC and its Thinking Process, teams can identify their core problems intelligently and they can then spend their efforts where it is most rewarding.

This can substantially increase their efficiency and velocity of the Value Streaming. We use the concepts of TOC during Restrospectives.

You can find a lot of good literature on TOC both in books and on the web.

Happy learning.

Venkatesh said...

Having been part of an organization that is migrating from traditional RUP to Scrum, I find the following interesting:
1. Knowledge Workers or Cheap Labourers:At the fundamental level, Team still does not believe they are knowledge workers. Especially, teams that have a bunch of junior members always hope that someone will guide them at task level. When this stops happening, there the problem starts.
2. Discipline:Product Managers want the teams to be flexible for change and believe if they adopt scrum, they can ask the team to do anything on any day. The truth is that Scrum demands more discipline. At least, new requirements or changes will have to wait till the next sprint.
3. Sprint Duration:Developers want longer sprints while others want shorter sprints. Usually the mighty will win... hence, sprints are way too shorter than what is ideal.
4. Motivation:Instead of motivating teams and appreciating and discussing about the product, managers are always on the lookout for 'data' to support individual appraisals. Now, how can this be transparent! The worry continues... The data such as individual bug inflow, outflow, number of features developed, etc are crap and managers are least equipped to make decisions based on these data (as to which developer is better than whom). The worst part is the Sprint champion award. Although, its not scrum that advocates this, this is another carrot on the hands of a poor manager. More often than not, these agile teams are not motivated - no wonder.
5. Peer Pressure:Scrum believes a lot on peer pressure. In practice, teams quickly understand that there are influential people who want to leave dominance on the team. These people (like product managers, people mangers, senior team members) have larger say on decisions and hence, peer pressure is subdued with hierarchical pressure on most occasions.
6. The Lazy Managers:There are always nice phrases to help the mighty... "Don't be bookish", "It's the need of the hour", "I know you can do it", "We must demonstrate personal passion by working longer hours", "He is the role model"... Come on, Managers, its high time, either stay out or dive in... don't do surface fishing.

Alright, everyone is still learning. Things will improve... but, it will be too expensive for the organization to lose its key self driven and motivated members in the name of agile migration without the presence of a proper influential coach.

Unknown said...

Scrum is a process, albiet a flexible process, however agile is not about process. Agile is process neutral. All it demands of a process is that it server the needs of the customer. If on a given project scrum can do that then use scrum, if waterfall will do that then use waterfall. Serving the customer in the best way possible is what drives agile, not the process.

Unknown said...

The success is based on strategic management actions and utilizing suitable tools that extend human thinking and describing the scope in a detailed enough level. Agile is a very good project manager software which i have been using for long time. Thanks for sharing your knowledge about this software.

Unknown said...

This is such an informative article and very clearly written. Every single thought and idea is direct to the point. Perfectly laid out. Thank you for taking your time sharing this post on Scrum is not agile