The workshop started with Kurt explaining the motivation behind organising it, after which followed the round of what Mark McKergow called “creeping-death” introductions. Although I also dislike such introduction rounds, it was fascinating to hear what diversity and depth of experience the attendees brought with them, and it was comforting to hear that they understood the same thing that I do for terms such as complexity and self-organisation.

After the introductions, we enjoyed the first part of a talk by George Rzevski, professor emeritus for complexity science at the Open University. I won’t go into the talk in detail; the slides are available on the workshop website. One slide, though, was very thought- and discussion-provoking. There is still no generally agreed-on definition of complexity, and dozens of attempts exist. George presented his 7 criteria for complexity, which we found that (although there were differing points of view that could be discussed) seemed to all of us to be ok, good enough, a good first step towards a common definition.

George Rzevski’s Seven Criteria for Complexity

1. INTERACTION – A complex system consists of a large number of diverse components (Agents) engaged in rich interaction
2. AUTONOMY – Agents are largely autonomous but subject to certain laws, rules or norms; there is no central control but agent behaviour is not random
3. EMERGENCE – Global behaviour of a complex system “emerges” from the interaction of agents and is therefore unpredictable
4. FAR FROM EQUILIBRIUM – Complex systems are “far from equilibrium” because frequent occurrences of disruptive events do not allow the system to return to the equilibrium
5. NONLINEARITY – Nonlinearity occasionally causes an insignificant input to be amplified into an extreme event (butterfly effect)
6. SELF-ORGANISATION – Complex systems are capable of self-organisation in response to disruptive events
7. CO-EVOLUTION – Complex systems irreversibly co-evolve with their environments

After lunch, we held an extended poster session. Having never done a poster session before, I simply drew one using visualisation techniques (thanks to Martin Haussmann for teaching me!), and was surpised at the positive responses I got. My topic is the bi-directional insights of using the Cynefin method to understand why Agile works, and using Scrum as a framework for managing work in the complex domain. I’ll be presenting my paper on this later this morning, and am looking forward to the feedback and discussions that will come. During the poster session, I also had interesting talks with Mark McKergow on solution-focused therapy and complexity, and with Ed Olson on his CDE model and the Cynefin ABIDE model, which I use quite heavily in my work on self-organising teams.

When the poster session was done, there was still some time until the evening barbeque, and George Rzevski presented the second part of his talk. He’s done fascinating and world-class work, I highly recommend checking out his slides.

Dinner, drinks and conversation followed, and by the time I got to bed, I was suffering from information overload, and was too tired to write this blog post, which is why I’m doing it now.

More later.

I was pleased because this acceptance is recognition of the research and work I’ve been doing for almost 15 years on understanding why Agile processes work. Even more, I was happy because my session on this topic, as well as my session on coaching self-organising teams, were both rejected for the Agile 2010 conference, although both sessions had been presented there twice in the past (as well as at many other conferences), and had received excellent reviews each time.

I’m starting to think that the average Agile practitioner’s position with regards to complexity science etc. can best be described as “a little knowledge is a dangerous thing”. Terms such as complexity, self-organisation, emergence, and so on, are bandied about in a way that they’ve become as content-free as the term “Agile” itself. People also seem to be content to accept such idiotic statements as “Architecture self-organizes around working code”. I don’t know about you, but the last time I saw architecture self-organising was last Friday night, when a re-run of Terminator 2 came on late-night television.

In any case, that’s enough for this little rant. I’m going to get back to my research and writing, and hope that some of you will be as interested as I am in finding out why this Agile stuff really works. More (including the finished paper) later.

http://www.cio.com/article/447712/_Agile_Leadership_Lessons_for_the_Suits
http://www.agilitrix.com/2009/12/coaching-self-organizing-teams/
http://www.infoq.com/presentations/coaching-self-org-teams
http://www.infoq.com/news/2008/08/coaching_teams
http://agilepainrelief.com/notesfromatooluser/2008/08/coaching-self-organizing-teams.html
http://chcrudy.wordpress.com/2009/08/30/joseph-pelrine-self-organizing-teams-and-turning-up-the-heat/

Now that’s what I call customer service! Thanks, Captain Martin Simon!

One of the reasons I met up with him was because I’m planning a small surprise party for another close friend, and I wanted to benefit from his experience and advice. What made Michael’s parties so good? During our chat, we realized that it was it the people attending, and their conversations. Not only that, it was also the music. Of course, we couldn’t forget the food and drink.

In the end, I decided to ask Michael to help me organize the party. He knew exactly what needed to be done for a party to be a success, and I was convinced that, if we just did the same things he always did, my party would turn out great. So, what was the first thing he said?

“Joseph, if you want to have a great party, you have to invite all of my friends.”

I thought I didn’t hear right. I wanted to invite my own friends, but on the other hand, I knew that with his friends, the party would be a smash, like his parties. Sensing my disappointment, he tried to cheer me up by saying that he could save me a lot of money. I wouldn’t need a DJ, he said, because he noted all the songs his DJ played, and he could just give me a playlist for my iPod. I’d have to lock the iPod away, so that no one could manipulate the song order, but he was sure that with these specific songs, played exactly in this order, the party would be a success. Also, it seems he videotaped his last party, and noted what everyone ate and drank. He’d provide me with scripts for all the guests, with precise timings, so that everybody ate and drank the proper things at the proper time. Of course, all conversation and interaction would also be scripted.

So, just imagine you plan to have a party, and it must be as good as the last one you held. You invite the same people, and just to make sure you’ve covered everything, you tell them to wear the same things, and speak only to the same people as last time. You’ve got the same music playlist, the same food, drinks and layout. Will this be a good party? Does something seem wrong with this story?

The answers to these last questions seem logical, although you can’t easily explain why. There is a reason, though, and this reason is key to understanding the basic issues surrounding complex problems and agile methods.

It’s called retrospective coherence.

One thing that makes complex systems complex is their causality. In an ordered system, if you do something, you expect a specific result. Do it again, expect the same result. It’s that simple. In a complex system, though, the causality emerges as the system emerges. As the party goes on, the reasons for its success become established, not before. After it’s over, you can say that a party was a success, and that these people were there, this music was played, and this much beer was downed  – but you cannot say that the party was a success because these people were there, this music was played, and this much beer was downed!

In a complex system, causality emerges as the system itself emerges, so that at the end, you can say how you got to where you are, but you can’t guarantee that by doing exactly the same things, you’ll get to the same place again – and you probably won’t. In complex systems, we say the causality is retrospectively coherent.

So let’s take a closer look, and consider what it actually was that made Michael’s party so good. It wasn’t any one factor, but how they interacted. It wasn’t the components, but the emergent complex system, that was a success. An interesting thing to note here is that the things that people often interpret as being the causes of a success aren’t that, they’re the symptoms. For instance, was it the people that made the party a good one? The music that was played? His choice of fine food and drink? Well, yes and no. Michael played a significant part because he’s a great host. He’s constantly walking around, looking, questioning. Is anyone standing all alone? Introduce them to a group! Does everyone have a drink? If not, get some more beer! Are people dancing? Trying to talk? Put on upbeat or down-tempo music as appropriate! All the time, the good host is surveying the scene and making small adjustments.

As a good host, Michael drew some boundaries. His bedroom was off-limits. His single-malt collection locked away. And, he set up a few things that people would be attracted to: a wide-screen television, a keg of beer, some games maybe. Every party needs attractors. Did you ever wonder why so many people congregate in the kitchen at a party, even if it’s the smallest room?
One thing he didn’t do is to define success, prescribe or proscribe behaviour (could you imagine saying that a party would only be a success if someone danced on the table with a lampshade on their head?). He let the party evolve, observed which patterns emerged, gently supported the good patterns, disrupted the bad patterns as soon as he noticed them, but generally let the party run itself.

Now shift the focus to a project. We tend to make the mistakes above when it comes to project planning. What worked last time? Why? Well, it must have been the people, the methodology, the meeting schedule – let’s do it the exact same way again. Like the repeat party, a project planned on this basis is likely to be a flop.

Contrary to Einstein’s definition, in a socially complex system, insanity is doing the same thing over and over again and expecting the same result!

(n.b. Dave Snowden has a classic story on organizing a child’s birthday party, and his story served as an inspiration for my version. You can find the story here. Thanks, Dave!)

Level 5: Burning
(results in team burnout and death marching)
At the burning level, the heat is too high. The stress level results in chaos, aggression, and a regression to primitive behaviour patterns. This is a dangerous level, since the high pressure increases the friction between the team members, increasing the heat even more.

Characteristic behaviour patterns for this level
•    Countless overtime hours leading to no result
•    Stress
•    Disagreements, fights
•    Sickness
•    Lack of communication
•    Inefficiency
•    Hectic
•    Blame
•    Restlessness
•    Self-preservation
•    Paralysis
•    Panic
•    Fear of attack

Level 4: Cooking
(ideal temperature for continuous improvement)
This is the optimal temperature for mixing up and forming teams. At this level, the heat is high enough to force disruption of behaviour patterns, but not so high that team members regress into pre-conventional mode.

Characteristic behaviour patterns for this level
•    Differences are resolved constructively
•    Continuous improvement
•    Enthusiasm
•    Work is fun
•    High productivity
•    Constructive discussions
•    Consensus
•    Joy
•    “We” – feeling
•    Self-reflection
•    Open discussions
•    Progress
•    Results
•    Creativity
•    Fun
•    High performance
•    Organised chaos
•    Good communication

Level 3: Cooling (Stagnation)
(discipline is lost and bad behaviour begins to fester)
If not enough heat is applied, or when things cool down after a while (entropy), a team enters the cooling or stagnating stage. In the kitchen, this is where what was once a fine-tasting soup has become a substrate for fungus and bacteria growth. Most health departments even have laws requiring cooked foods to be cooled down within a certain short period of time, as to pass through this stage as quickly as possible.

Characteristic behaviour patterns for this level
•    Slacking off
•    Work becomes routine
•    Loss of discipline
•    Monotony
•    Indifference
•    9 to 5
•    Too much discussion
•    Minimal communication

Level 2: Congealing
(team gets too comfortable to achieve and bad habits become the norm)
Increasing pressure to adhere to “norms“ – “that’s the way we do things here”

Characteristic behaviour patterns for this level
•    Change becomes difficult
•    Many meetings
•    Multiple implementation of work
•    Problems become difficult to solve, because everything’s a mass/mess
•    Boredom
•    Much talk, no action
•    Consensus becomes difficult
•    Resignation
•    Splitting of the teams into cliques
•    Everyone works for themselves
•    Minimalism
•    Defensive stance
•    Hiding behind procedures and rules

Level 1: Solid/frozen
(control takes over and change is no longer possible)
Inflexible, rigid burocracy – “that’s the way things are done!”

Characteristic behaviour patterns for this level
•    No interest or excitement
•    No initiative
•    No exchange or communication
•    No motivation
•    Standing still
•    CYA
•    Feeling of Powerlessness
•    Retreating to and hiding behind rules
•    Avoiding work
•    Lack of motivation

A topological observation of the heat model

An interesting observation, and one which may help explain the entropy analogy, is the fact that the characteristic behaviour patterns seen when the system cools down to the solidifying level and similar to the ones seen at the burning level. The system seems to wrap around topologically (spiral?), similar to the bottom section of the Cynefin butterfly model.

In the “turning up the heat” model, we differentiate between 5 different levels of heat: burning, cooking, cooling, congealing, and solidifying. It takes effort and energy to maintain any level of heat. The dissipation of heat in teams can be thought of as ‘social entropy’.  The natural forces of entropy as described in the Second Law of Thermodynamics apply as much to social organizations and interpersonal relationships as they do to molecules or galaxies.

Why do teams, even ones that are experiencing problems, not want to change? My thesis is that most of the time people and teams are in stasis and so resistant to change. To make change occur we have to raise the heat just enough people no longer feel comfortable in their current environment. I believe that when we see teams that stop doing TDD or other practices that it’s often because the heat has been turned off. Working in a rigorous fashion I recommend that we only make one change at a time in response to problems we want to improve. This allows us time to observe the system for retrospective coherence and adapt accordingly. It also reflects the fact that changes happen with a delay, and helps us to avoid over-correction (see Dörner’s example, later in this chapter).

The use of heat has analogy in cooking – most people on turn heat on or off but as coaches we need to understand the five levels. To connect the analogy:

Burning – food tastes burnt, teams fall apart.
Cooking – flavours in food are well integrated, teams adapt to new ideas
Cooling – bacteria grows in food, teams stagnate and start to stop using tools
Congealing – teams are starting to lose their flexibility and lock in their habits
Solid/Frozen – bureaucracy has set in, there are forms to fill out and sign offs

A coach needs to recognise the characteristics of a team in each of the stages. Most often, a team will be moving between levels, unless they’ve got to the state of complete burnout or complete stasis.

In the next book excerpt, we’ll look at the individual levels in more detail. It’s challenging for me to explain them, and I find that they’re most easily defined by example.

Last week, I attended the first Swiss Lean/Agile/Scrum conference. As usual, the Swiss take a long time to catch up with new ideas and technology, but when there’s money to be had, they’re right up at the front of the line. Anyway, Ken Schwaber gave a very interesting presentation on the concept of “Done”. He showed the canonical burndown chart, and then took the individual vectors apart.

Seeing the variables separated like that got me to thinking, and I had the chance to run my thoughts past Karl Scotland and Keith Braithwaite, both of whom went out for a beer with me afterwards.

Back in the early days of XP, we defined a set of project variables:

The rule went, “Time, Resources, Quality, Scope – choose three”. Whichever three you chosse, the fourth variable would be a function of the other three. Which three were actually chosen was the customer’s decision. Some customers (and managers) don’t understand this rule, and try to grab control of all 4 variables. By doing that, the first variable that’s dropped is quality, followed by time.

So, how do these sets of variables map to each other? The first two are easy:

• time = time
• backlog = scope

The other 2 are a bit more difficult:

• V = f(R)

Velocity (burndown rate) is a function of the available resources. Regardless of what you have to do, having more resources will normally allow you to go faster. This function is non-simple and non-linear, unfortunately: as Fred Brooks rightly says, “adding manpower to a late software project makes it later”, but the simple case is a more or less linear function.

Quality is even more complicated:

• q = ∂V/∂t

As Dan Rawsthorne says, “Quality is the first derivative of the burndown”. The quality of a product is directly related to the development velocity/burndown rate.

So, so much for that thought. Keith spun the thought even further, proposing in his recent blog post that you can increase velocity by increasing quality. I agree with him, and I recommend you read the post.

A very interesting incident took place at the gathering, one that got me thinking about the practical application of my work in social complexity to the people and teams I deal with. The gathering was structured as an Open Space, running all day Saturday, and all 40 participants met up Friday evening to discuss and decide on topics for the next day. At 19.30, we had a full board of interesting topics, Rachel announced that the evening session was over – and then it happened.

40 people sitting in a room, all thinking “now what?”

My friends know that I hate the tradition of everyone at a conference going out to eat together. You all pile into some poor restaurant, start pushing tables together, and generally drive both the restaurant personnel and the other diners mad. Why do you need to push tables together? It makes it much more difficult for the waiters and waitresses to manoeuvre, and honestly, you can’t talk to more than the 5 people surrounding you at most. It’s loud, it’s noisy, it’s uncomfortable. It puts a stress on the waiters and waitresses, taking all the orders, serving them all at once, and then having to deal with people demanding separate checks and paying with a mix of cash (in multiple currencies), debit card, and credit card. It’s even worse for the kitchen staff, who get bashed with a large number of orders without any warning.

40 people sitting in a room, all thinking “I’m hungry.”

As if the problem with the restaurant wasn’t enough, we had to deal with people staying in a number of different hotels, none of which were within walking distance from the gathering site, and a number of people who came by train, and had no way to get to either the restaurant nor the hotel.

So, how does one solve this problem?

Rachel gave it a good try. She told me afterwards that her prime concern was to have no one left stranded. Once she asked people who needed a ride somewhere, though, things took on their own momentum, and there were cries of “everyone who wants to eat Indian in this corner” and “anybody staying at the Premier Inn in this corner” and “anyone with a car over to this side”. Pure chaos. Admittedly, Rachel and Mike had tried to plan things a bit, and had posted a list of restaurants for people to choose from, but as you’ll see if you look at the list (http://tr.im/mqen) some people either couldn’t/wouldn’t decide, or were gaming the system by keeping their options open and not committing themselves.

“So”, I thought to myself, “this is an excellent example of what happens when you try to apply ordered systems techniques to a complex problem domain. How would I deal with this?” And so I used the incident as a case study in my Open Space session on self-organization the next day.

We all tend to fall into the habit of trying to bring un-orderly situations under control. In complex situations, however, even someone as skilled and experienced as Rachel had a tough time doing so. The domain had three main variables – restaurants, hotels, and transportation – with each person having an unknown combination of the three. With 40 people, that’s a lot of possible combinations to match up.

So, how would I solve the problem? Instinctively, my first intervention would be to “turn up the heat”, to give people the sense of urgency necessary to break through their lethargy and self-organize. Saying something like “we gotta get out of here in 5 minutes, because they’re closing” would provide the shock necessary. The problem here is that the level of heat generated is not predictable (we’re in a retrospectively coherent complex system here), and we don’t have the luxury of doing multiple probes in a probe-inspect-adapt loop. Would 10 minutes be better? Would 3 minutes suffice? I don’t know, so it’s time to start thinking about what I do know.

One thing I know – or rather, assume with a high degree of certainty – is that all the attendees are adults. They got to Bletchley Park., which means they took responsibility for organizing that, and they assumedly took responsibility for organizing their accommodations for the night. It’s 19.30, and everyone’s there, so no one has a hotel that they need to check in to immediately, lest they lose their reservation. So, why not just forget the hotels for now? Doing that would bound the problem more by reducing the number of variables and complexity. People can go to their hotels after dinner. Also, letting people go to dinner first works to our advantage. Social network stimulation takes place at dinner, people find others staying at the same hotel, and the restaurant staff are usually knowledgeable (if not always helpful) about the relative locations of the restaurant and the hotel. They’ll know how far a walk it is, and can also call a taxi if need be.

This leaves us with two variables  – food and transportation. Given a choice between catching a ride with someone, and going with interesting people to a restaurant that might not have been first choice, or being fixated on a particular restaurant, at the risk of calling a taxi to get there, and then having to eat alone, I assume that most people would choose the first option. This assumption is supported in part by the fact that quite a number of people signed up for multiple restaurants during pre-planning (see above). So, with the restaurant variable becoming at least partly dependent on the transportation variable, we’ve gotten things down to one variable – transportation. The simplest way to solve the problem of one variable would be to ask all those attendees who have no transportation to raise their hands, and ask the others to help them. Wait around for a few minutes to ensure that everyone’s been taken care of, and no one is standing there alone and lost, and you’ve solved the problem. You’ve also solved it in terms of Rachel’s initial concern, i.e., that no one was stranded.

What did I do? I had driven up to the meeting with my good friend and business partner David Harvey – actually, he drove, since I can’t deal with people driving on the wrong side of the road. Knowing our mutual dislike of large crowds in restaurants, we checked around on the net for a nice quiet place that serves good food, and that was a bit out of town, and booked a table for four there. As we anticipated, we weren’t the only two who wanted to avoid the post-meeting turmoil, and so we spent an enjoyable evening eating and chatting with Steve Freeman and Keith Braithwaite.

Vegetable soup

My grandfather was a fantastic chef, and he was a great inspiration to me. My parents named me after him, and not only did I inherit his love of, and talent for, cooking, but unfortunately also his girth. One of the first things I learned from him, at a very early age, was how to make vegetable soup. OK, maybe he just wanted some cheap help for mise en place, but at that age, I didn’t care.

Vegetable soup is simple. It consists of carrots, celery, onions, leeks, some herbs, and lots of water. That’s all. He’d let me peel the carrots and onions, and then he’d chop up all the ingredients and throw them into a pot that was so big it reminded me of something the cannibals threw the missionaries into in an old movie. An hour or so later, magic had happened, and out of all these ingredients emerged a tasty soup.

One day, after I was old enough to be allowed to use a small knife myself, I decided to surprise my grandfather and cook vegetable soup myself. Early one morning, I snuck down into the kitchen, got the ingredients, prepped them, then threw everything into the pot and waited for my grandfather to come. When he entered the kitchen, I jumped up, ran over to him, told him I had a big surprise for him, and proudly pulled him over to the stove. He looked into the pot, and what he saw was – mush. Careful to not upset me, my grandfather said that I had forgotten something. “But I put in all the ingredients as you always do,” I said. “Yes,” he said, “but the most important thing in making soup is not an ingredient itself – it’s turning up the heat.”

The same thing happens to us every day. We get a group of great people together, expect they’ll make a great team, and then watch as they act like a bunch of idiots, without understanding why this is happening, or what we still need to do to get them to work well together. We’re forgetting the main ingredient!

The team is the fundamental unit of software development. A team of skilled and highly motivated individuals, creative, focused and productive, is the cornerstone of the Agile philosophy. It does not follow that a group of people, however carefully selected, will auto-magically form a team capable of producing excellent software, on time and within budget.

Any group of people will self organize, according to the social and psychological principles, or ‘human nature’ that have influenced human interactions since the days of cave dwellers. Self-organisation will happen eventually, then, but there are two reasons why leaving a group alone to self organise for software development will almost undoubtedly fail. First, few projects incorporate the time or resources necessary to allow a stable organization to emerge before any work is done. As the saying goes, ‘a drop of water may hollow out the hardest stone,’ but we can’t afford to wait that long. Second, the team that emerges will be likely to have developed according to individuals’ ingrained behaviour patterns, many of which will reflect our primate ancestry. All groups of primates self-organize. This mainly results in the establishment of a pecking order, a rank and power hierarchy. Self-organization ends up being a fight for alpha male dominance. This what probably not what you want – unless you’re the alpha.

So, for a group to become a team, you must turn up the heat high enough that the team members cannot maintain their ingrained behavioural patterns.

If you don’t increase the heat, people won’t need to change the way they normally behave, and so they won’t change. Any self-organization will revert to being the basic primate fight for alpha male dominance. Unfortunately, this isn’t always an easy thing to do. Another thing I learned from my grandfather: for most people who haven’t trained as cooks, heat is a Boolean. It’s either all the way on, or all the way off. A good cook, however, knows how to regulate heat, how to use it to his advantage. He also knows which kind of heat to use in which situation. Too much heat, and things burn. Too little, and they don’t cook, don’t blend well enough to become soup – or whatever else you’re cooking. No heat at all, and they stay in their primitive state – raw. Only when the heat is right does cooking happen, do flavours blend, whilst retaining nuances of individuality.

It’s the same thing with people. Too much heat, and they burn out. Too little, and they don’t feel the need to change and adapt. Only when the heat is right do they adapt their behaviour and meld together to form a team.

As the posts go on, we’ll look at the Heat model, its different temperatures, the thermometers (analysis tools) available, and the various stoves (interventions and techniques) available to turn up the heat on a team.