Social Complexity and Agility

The Volcano Principle

(n.b. Back in 2011, I was asked to write an article for a German magazine for their special issue celebrating the 10 year anniversary of the Agile Manifesto. I wasn’t too happy with the direction their editors pushed the article in, and since a lot has happened since then, here’s a new, English version).

Responding to change

On 14 April 2010, a volcano in Iceland exploded. As we struggled to pronounce “Eyjafjallajökull”, air traffic throughout Europe was paralyzed by clouds of ash. I was lucky enough to be trapped at home, much to my client’s frustration, but others had it worse. Friends of mine had to travel for 48 hours by train to get home from a client’s site, and I even heard of a stranded Swiss Air pilot who missed his own wedding.

I call it the volcano principle. Time and again, things happen that are beyond our control, but that affect us. Things that we have to respond to. We do not know when something will happen; we do not know what will happen. Some of us do not want to admit that such things happen, and are then taken by surprise by the events. Others are Agile.

We value: responding to change over following a plan

As Harrison Owen says, there are no closed systems (see [Owe08]). We never know when and where volcanoes, which we may have never previously encountered, will erupt. It may be that our customers have new needs and wishes. It may be – and it is often the case – that our competitors bring something new onto the market. Or it may be that politicians adopt new laws or regulations that we have to consider. Volcanoes lurk everywhere. Either we react to them, or we are out of the game.

How much are you prepared to throw away?

When I worked with Kent Beck in the early days of the Agile movement, we spent time trying to explain to people what the basic and fundamental ideas behind eXtreme Programming (XP) (see [Bec99]) were. Kent often used the metaphor of driving a car. That was entertaining, not only because Kent could tell some horror stories of his first driving lessons with his father, but also because driving was suitable as a metaphor for many XP ideas, principles, and techniques. Driving is an example of test-driven development – the tests are your eyes. It is an example of cost estimation – how far is it to the destination, which is often answered as time and not distance. It is also a good example of quickly adapting to the current circumstances – when driving, you are not merely pressing on the accelerator, but always correcting – a little faster, or slower, a little left, a little right. The driving analogy shows up the issues that arise when you’re stubbornly following a plan, or directions. Once there is a detour, a traffic jam or an accident, you’re stuck if all you have are directions. The phrase “responding to change over following a plan” reminds us to, demands that we really focus on what’s important: that we do not stubbornly follow a plan, but that we reach our goal. Agility requires of us that we take to heart the following question: what do we want – the implementation of a first specification or a successful project? The two will never be the same.

We can even go further. One of Kent Beck’s favourite sayings was: “How much are you prepared to throw away?” If we think about driving a car, when we have invested time in developing a plan or getting directions to a far-away place, we will often cling to the plan – even when it is no longer useful or if the road is not passable. Otherwise, the time we invested would be considered as wasted time. The Agile approach, however reminds us that there will always be sites or volcanoes – so we shouldn’t consider them to be exceptions processed by change requests. Instead, we should Embrace Change, we should expect from the beginning that changes will take place, and should build our process accordingly.

Where are we now?

But where we are today, ten years after publication of the Agile Manifesto? Do we really live those values, or do we just go through the motions? My friend Dave Snowden describes the development of new methods in his inimitable way, as follows (see [Sno05]):

• An Academic group studies a range of organizations to identify causal linkages between things those organizations do and results that they achieve or fail to achieve, from which they derive a hypothesis that forms a definition of best practice. A popular management book then follows and a new “fad” is born.
• Consultants and IT providers produce industrial-strength recipes based on the new idea, which ideally involve a consultancy process, followed by a technology implementation, and some form of organizational change or cultural alignment with the programme to orient employees to the new goals
• Managers go through a process based on the recipe to determine a desired end state defined in terms of economic performance, behaviour characteristics etc. They then determine the current state and identity a series of process steps to achieve that goal and roll out the programme promising substantial improvements as a result to their stakeholders many of whom in the “employee” category are already suffering from substantial ‘initiative fatigue’.
• Some years after the fad has run its course in industry, and its limitations are apparent, the consultants find a lucrative secondary market in applying “industrial best practice” to government clients.

Back when Kent Beck first published his ideas about a way we could develop better software, he was often laughed at. Many people thought that something like this could never work. In this sense, the first edition of “Extreme Programming Explained” was perhaps not the great management book, but it was a call to battle against old methods of software development.

XP is not easy. At the time we joked, for example, that we do pair programming because no one has the discipline to stick to all the XP practices alone. The first consultants and IT providers who worked with XP, like Ken Auer and Joshua Kerievsky in the USA, or Steve Freeman, Tim Mackinnon, and the other founders of London’s eXtreme Tuesday Club, had to go on a journey of discovery, and they allowed their clients to share their experiences rather than selling them recipes or methods.

Then came Scrum

Some causal linkages between the things organizations do and the results that they achieve or fail to achieve were described in a 1998 paper on Scrum (see [Bee00]). The patterns described there refer in turn to an article published in 1986 in the Harvard Business Review (see [Tak86]). The authors have, however, discreetly circumnavigated the problem described in Snowden’s statement, “Patterns have a predictive capacity only within clearly defined ontological boundaries” (see [Sno03]). The domain of patterns as “Best Practice”, that can be applied repeatedly, in different situations, to obtain reproducible results, is in simple and complicated systems, which are inherently ordered. But in complex systems, of which software development is one, a retrospectively coherent causality exists (see [Pel09]). Viewed like this, it should come as no surprise that those “best practices” don’t always work.

The requisite management book came from Ken Schwaber (see [Sch01]), who also supplied the associated branding and the money machine: the “Certified ScrumMaster” Program. Whereas XP was “extreme” also in the sense of being hard to do, Scrum offered a lower barrier to entry. You didn’t need an understanding of software development, let alone to be able to develop software yourself. “It’s only management,” thought the many management consultants who jumped on the Scrum bandwagon. This mixture of a number of “best practices,” a management book, and a low entry threshold, meant that the way was open for the consultants and IT service providers. They duly arrived with their “Scrum Checklists”, but their primary goal was often to market their own tools and methods as Scrum (compatible) in order to milk a new cash cow. Is it any wonder that Ken Schwaber says, “75% of companies that say they’re doing Scrum will never achieve the full benefits”?

Just as the star of Scrum was starting to fade, the next Silver Bullet arrived: Kanban. The same consultants who told us a few years ago that XP was too heavy, and it would be easier to get started with Scrum, now say that Scrum is too heavy and requires a reorganization of the company. Just put up a Kanban Board, limit the work-in-progress – and magically everything becomes visible, and you’ll be fine. The approach is good, but it is not enough. It is not enough merely to show people where their weaknesses are, as some are remarkably resistant to change (see [Keg09]). This transparency is a necessary, but not a sufficient, condition for change. Just as Scrum – used properly – showed that the technical practices of an XP approach are required to deliver high quality software, Kanban – done correctly – will show that certain practices from Scrum and/or XP make sense, or are necessary if one wants to continuously improve. (n.b. the question of whether Kanban can or should be classified as a fad, according to Snowden’s criteria, is left as an exercise to the reader).

Sure, there are world-class companies such as Joshua Kerievsky’s Industrial Logic, who work without effort estimation, planning, iterations, and so on. Joshua, though, was one of the first pioneers of XP – someone who has gone through the whole learning process from the ground up – and his team have worked together for years. Such a world-class team will be successful in spite of each method. Unfortunately, most companies that want to throw everything overboard are not nearly as far advanced or as good as Industrial Logic. “It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change” (Charles Darwin).

What about us? Are the players in the Agile Scene (coaches, trainers, Scrum Master, Scrum Alliance, change agents, etc.) Agile themselves, in the sense that they embrace change? Should they? How should the players themselves deal with this demand, this challenge?

Scrum, for example, is a very good, simple framework for managing work in complex environments. Scrum is Agile, adaptable, and anything but prescriptive. This is precisely why it requires an understanding of the underlying theories and principles to get it right and to apply it in sustainable manner. The work around this basic understanding has been rather slow, and only a few researchers are concerned about the deeper reasons why Agile processes actually work (see [Pel10]). Many ignore the fact that science has made big jumps since the publication of the HBR article (see [Tak86]) and Scrum pattern-paper (see [Bee00]), possibly also because the latest research critically questions some basic assumptions of these papers (see e.g. [Gou06] and [Nor06]). Instead, there is now an open struggle going on between the Scrum Alliance on the one hand and Scrum.org on the other side about who “owns” Scrum, and about what Scrum “really” is. These attempts to nail down the definition of Scrum, together with efforts to develop certifications based on these definitions – the battle for the “one true Scrum” – are leading to the problem that Scrum is being defined more rigidly and prescriptively, and is itself becoming less Agile.

Where people are Agile is in finding ways to make money from the Agile methods. Although we all have to earn a living, we must never forget the first principle of the Agile Manifesto: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Maybe the manifesto is missing a sentence such as: “We value helping our customers over taking their money.”

Outlook

What do I think will happen in the second decade of the Agile Manifesto? I dare venture only a few predictions: research will surely continue, the search for better ways to develop software will too, as will the search for Silver Bullets and new sources of earning money from Agile. I am confident that new and better methods will appear. But even if they do not, and if only the last point of Snowden’s description holds true, and the current “industry best practices” take hold in government, they would constitute a step forward.

I close with a guiding principle of the first Agile philosopher, Heraclitus: “The only constant in the universe is change.”

Literature & Links
[Bec99] K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley Professional, 1999
[Bee00] M. Beedle et al, Scrum. A Pattern Language for Hyperproductive Software Development, Pattern Languages of Program Design 4, Addison-Wesley, 2000
[Gou06] S. Gourlay, Conceptualizing Knowledge Creation: A Critique of Nonaka’s Theory, Journal of Management Studies, Volume 43 Issue 7, 2006
[Keg09] R. Kegan, L. Lahey, Immunity to Change: How to Overcome it and Unlock the Potential in Yourself and Your Organization, McGraw-Hill Professional 2009
[Nor06] D. Nordberg, Knowledge creation: revisiting the ‘ba’ humbug, 2006, see http://ssrn.com/abstract=891068
[Owe08] H. Owen et al, Leadership for high perfomance in a self-organizing world, Berrett-Koehler, 2008
[Pel09] J. Pelrine, On retrospective coherence, 2009, see http://bit.ly/95Pqqt
[Pel10] J. Pelrine, on understanding software agility – from a social complexity point of view, 1st Int. Workshop on Complexity and Real-World Applications, Southampton, UK 2010
[Sch01] K. Schwaber and M. Beedle, Agile Software Development with Scrum, Prentice Hall 2001
[Sno05] D. Snowden, multi-ontological sense-making, Management Today Yearbook 2005
[Sno03] D. Snowden, Managing for Serendipity: why we should lay off “best practice” in KM, ARK Knowledge Management, Vol 6 Issue 8, 2003
[Tak86] H. Takeuchi and I. Nonaka, The New New Product Development Game, Harvard Business Review, Jan-February 1986

Leave a Reply

You must be logged in to post a comment.