Full Stack Systems Thinking

Full Stack

I’ve figured out a name for what I do. It’s Full Stack Systems. It’s important to have a name for things, so we can notice them, and can choose to notice them. I’ve borrowed Full Stack from the IT term Full Stack Developer, meaning someone with the skills to program back end and front end systems, and who understands the full delivery model of their work.

Full Stack Systems Thinking looks at the connections between things as much as the things themselves. It looks at patterns, emergence, interconnectedness and other systemic stuff.

Going up the stack

  • Knowing Myself, knowing the patterns I use, the internal dramas I have
  • Understanding the way groups interact at their best and worst, and how they can work in curiosity or contempt
  • Challenging my own and others Theories in Use and Theories in Practice
  • Seeing how groups interact, and the patterns and drama between them
  • Choosing to understand different decision making contexts
  • Being curious, helping and bringing others along
  • Applying theories of organisation to individuals and groups
  • Recognising that behaviours are structurally coupled

I also feel the this also involves

  • Working Visually
  • Choosing to work and share freely with others
  • Always getting more than one perspective
  • Moving towards what is difficult

I’m not the only person doing this, and others already have names for their work. Full Stack works for me.

Full Stack Work is a type of generalist. It’s important to say it’s not better than any sort of speciality. In a given situation it may be more or less appropriate than a specialist approach.

Full Stack Work is slow to learn, it moves on wide front, and it’s always less deep, but more connected. This may be what you need and it’s what I do.

I’m recognising others, specialists and generalists who are along for this ride. Connect on twitter.

Anti-Fragile You

Before we begin, I prefer to be interesting to being correct. It’s a sort of anti fragility, yeah?

OK let’s go.

What is Anti Fragility?

Anti Fragility is the property of things that become stronger with disturbances that could be expected to do damage, be absorbed or shrugged off. Example would be a rock, a china cup and a muscle. Rocks just absorb stress, muscles get stronger with the right amount of stress, nothing good comes from stressing a china cup.

Anti Fragile People

Humans have a lot of anti fragile properties. Just through living we’re crawling up the ramp against entropy – and while a broken bone won’t repair itself to be stronger than before, certain types of activity that would create wear on a machine, creates growth in us. Humans are not machines.

So in general bad stuff happens and we either end up worse, not bovvered, or we get stronger after getting weaker. There are activities and exchanges that are created by others to grind us down, things that just seem to align to grind us down, and the person who gives you a break, pointing the way to success out of a defeat.

I was expecting to write a list

Can we choose to engage in activities where we’re likely have the opportunity to learn from a potentially damaging failure? I had expected to write about all the things you could choose to do to be anti fragile. Lots of articles link being anti-fragile to being safe to fail – but it’s a lot more than that. To be anti fragile, you first need to be fragile. This hurts, it may not be fun, torn muscles, failed plans, stuff you don’t necessarily want for an easy life. Our brains will run acrobatics to make sure we’re the hero in our story, to make sure that we’re consistent, and it’s not us who needs to learn.

After starting a list of “things to do to be anti fragile” I realised the things on it were not really anti fragile. They seemed to be repeating the mistake of thinking anti fragile is just not putting all your eggs in one basket. Safe for things to fail, as long as one thing works. Like having a diverse portfolio, hedging your bets or just making sure you’re robust to total failure and using any excess resources to try to hit success.

Robustness confused with anti fragility

Anti fragility need fragility first. For exercise, running a marathon can kill you, and usually makes you really hurt. Lifting weights badly can permanently damage you. But exercise is also addictive, and done in moderation helps us become stronger, and take some small wins from entropy. Exercise is probably the most popular anti-fragile activity humans engage in.

We don’t even need to know this to exercise. We intuitively know we get better at running by running. And we get exercise happy drugs produced in our brain. But mental anti fragility is not the same. We really don’t like being shown how we’re wrong. We get “mental anti fragility feel bad drugs” instead that kick in enabling mental acrobatics of the “I’m not wrong, they’re wrong” type.

I wrote about “double loop learning” this time last year. Double loop learning is sometimes used to mean needing to figure out a new way of doing something. I think it’s more than that. It’s realising that your current knowledge is wrong, and is responsible for the problems you have. Anti fragility is getting out of this bind.

The paragraphs below show how opportunities to be anti fragile could show up. Opportunities to be mentally anti fragile don’t have to be taken. They are unlikely to be the easy option.

From “Metaphors in Mind” by James Lawley and Penny Tompkins p185

Onwards to the Lists

Here are the lists I’ve made.

Starting off with a list of things that could make you personally fragile

  • People questioning your beliefs
    • Often without stating theirs
    • Trying to make you angry
    • Making you do their fact checking
  • When you’re pitching your ideas without getting feedback on success or failure
    • You don’t know why you failed. You can’t improve
    • You don’t know why you succeeded. You dare not change.
  • When you have less Information, or an information imbalance
    • Getting in a law based dispute with a lawyer
    • You may be right, but they know the loopholes
  • A power imbalance
    • Like arguing on twitter with someone who has loads more followers than you
  • Being in debt
    • Not much good can happen here as an individual
  • Being a whistleblower
    • Again, not a successful career move
  • Being asked too many questions
    • Mental overwhelmed-ness
  • Being asked to explain something you’re learned to do automatically
    • Asking “how do you do that weird thing when you take penalties” to someone just before they step up to take a penalty.
  • Being hooked onto drama and the following brain state
    • Getting triggered, angry, and not being able to think
    • Half of the political establishment in the USA may be feeling this right now

So anti-fragility comes after these things, and because of these things. Anti-fragility means using what has harmed you to make you stronger. What might that look like?

  • Know your beliefs
  • Choose only work where you get feedback on success
  • Choose to play an infinite game
    • You play to continue playing, not to win
  • Recognise information imbalances
  • Recognise hooks onto drama

Hooks onto Drama

Learning to recognise hooks onto drama is interesting to me. Hooks onto drama are those conversations where escalations away from where understanding or learning is likely to occur.

Like in a job interview where you’re asked

Tell me a time when you have to solve a complicated problem?

And then, when you’re flushed with stories of success being asked

So, whose fault was that?

And you start to blame and persecute, because someones asked whose fault it was, and it would be rude not to continue the conversation in their chosen direction.

Anti Fragility has a Posse


Like hooks onto drama need someone else to wield the hook. Someone needs to do the mental heavy lifting to give you the opportunity to learn from fragility.

What would an “anti fragile you” look like?

Who helps you?

Who hooks you into drama?

What has to happen to make an “anti fragile you” happen?

This post was helped along by conversations and workshops at Northern Taste of Clean 2018 and Reimagine Work 2018. Thanks to everyone.

X Marked the Spot

Photo by Jeremy Bishop on Unsplash

The innovation should be right about here.

The world is changing fast, and some of the organisations doing the changing have insights that the rest don’t. Visualisation of information allows people to point at a model outside of their head, ask better questions and make better decisions. Wardley  Maps are a great way to visualise what happens to provide value to someone in a way that makes for better questioning about why. This post is about the kind of questions that we may like to ask to create the change.

I’m releasing v1 of this post. If you spot any typos or missing words let me know in the comments please.

What is mapping?

I’m using Simon Wardley’s “Wardley Maps”. We show at how a user need is met through a value chain on the y-axis. We use the x axis to show how we source the components in the value chain, from the left to right.

An example map from Simon Wardley is below. There are two Users, and the need A also requires other elements. From this map we can see that we choose to build the things in Red, we buy or rent the things in blue, and the things in green are commodities.


CC 3,0 Share Alike

Sourcing the components ranges from

  • inventing your own
  • building something unique
  • buying a product or service
  • renting a service or commodity

Over time the way we meet user needs moves to the right. In the map below the way to build a website has moved through these domains. For simplicity,  I’ve not added elements further down the chain, and the y-axis is far less important than the x-axis here.


If you need more information about Wardley Maps, this post explaining the Wardley Mapping Canvas by @benmosior is a great intro

Once you can see where you are sourcing all of the parts of the value chain from you can ask “Does it still make sense to build or buy the thing here?”. In the above example, you may have employed a Computer Science graduate in the 90’s to write HTML code, until you could buy a product to do that for you, and pay less to employ a Web Designer. Those who wrote code by hand claimed to write faster and better sites, but eventually, they needed to get jobs elsewhere. Similarly, in 2018 people with Web Development Package Skills are replaced by DIY efforts currently on Square Space or Wix.

This movement over time is a central part of Wardley Maps.

User Needs

Over time the above map follows Gartner’s hype cycle. New things get invented, they need some work to build products that people are happy using, and then if they cross the chasm more industrialised efficient products and services emerge.

The products and services meet a user need that is used to anchor the map.

Everything that is successful, that once made sense to build yourself, becomes a product. While there are various strategic to stop things being taken and improved by others, successful companies will need to alter the things they build in-house if they want to compete with companies that arrive later. In this case, Amazon disrupted its book business with e-readers, and Costa Coffee produce drinks from an automated machine in many petrol stations, with little of the Costa experience.


I’m needy.

Wardley maps anchor on User Needs. These differ depending on the persona. For example, for a children’s party you can source a totally different value chain depending on the type of party you want and can pay for. In this Map you can see the high-level requirements for cake, entertainment and location. You can source the items for your party from anywhere, but you’re unlikely to have a unique party with a cake from Sainsbury’s, or at an established children’s party venue.

There is a market for everything in the map, and the provisions move to the right. For example, you could invent and build your own one-off children’s party Escape Room. If it’s a potentially profitable idea then there will be some artisan companies making them commercially. Then bigger entertainment companies may start to make brand name cookie-cutter offerings. These will be cheaper, and meet a need, but not the same need as the artisan ones. If your need is ‘A party like no other’ you’ve moved on.


There is the ability to make money providing service at each part of the x-axis, however, over time those things that are successful will move to the right as they are produced more efficiently and become products for general consumption. Then, they have a slow entropic death 🙂


As companies develop their product they find ways to build thing more efficiently. Incumbent suppliers, who have not needed to look for efficiencies are prone to be disrupted by “Digital” companies who are able to take the element of the value chain and move them to the right (or remove the requirement altogether) using innovative IT, Cloud Computing and Lean Thinking. Canonical examples of disruption are Kodak, Retail and Blockbuster, but Banking and Education are also targets. Anywhere where the reason for sourcing part of the value chain is “we’ve always done it like that” is potentially disruptable.

Previously successful companies are currently trying “Digital Transformations” to try to keep competitive with new companies that do not have their legacy. The new companies will be disrupted eventually. Lots of consultants are trying (and largely failing) to digitally transform companies. Interestingly many use tools that are also moving right on the Wardley map, (roll out the transformation, yeah?) and this is key is understanding the failure. More in a future post…

I didn’t know I liked that bit!

It’s worth mentioning that the new way of meeting a User Need will be different from the old. There may be things that customers appreciate that they may not realise at the time of disruption.

  • Books deivered to the door – But I liked browsing bookshops
  • 10000s of digital pictures – But I liked sitting as a family looking at old photos
  • Packages delived by drone – But Posties also keep an eye out for vulnerable people living alone
  • Educated online by a MOOC – But I liked moving away to University and creating deep lifelong friendships, and going out every night.
  • Good coffee available everywhere – But I like an artisan coffee shop experience
  • I like books – I’m not even buying a kindle….

So there is life after industrialisation for certain artisan pursuits. You can still buy axes to chop wood and hipsters will press your coffee through an Aeropress by hand.

User needs are also created – feeding our impulsive monkey brain, our need to belong to a tribe or to feel good about ourselves etc. Most strategic play, however, is based on cost. But it doesn’t need to be. Also creating the user needs is part of Strategy not on the map, people don’t just want a new thing for no reason.

How not to be disrupted

There are techniques to not be disrupted. These include patenting your ideas, creating a strong brand and talking down your competition. Linux is cancer by Steve Balmer was an attempt to stop Linux moving operating systems to the right and commoditising them. Many other methods to stop operating systems being commoditised had been tried by Microsoft and failed.

Sometimes, providing an experience by doing some things way more inefficiently than they need to be done provides the user with what they (now) need – Bespoke Apple Stores, for example, make the Apple experience harder to disrupt.

Another example is retooling your factories with people instead of robots so you can sell bespoke customisations to your customers that your competitors cannot offer. Your competitors are too industrialised  with robots to compete. See my post about Mercedes proposing to replace robots with people. They are keeping their product in the profitable centre of the map.

Know your place

What part of the x-axis are you good at? There are different approaches, skills, mindsets, vocabularies and metaphors in use at each part of the x-axis from building something unique to managing a rental product or commodity.


These guys focus on doing one thing well

The team above are amazing at making whatever it is you’re doing super efficient. Even the one getting a different perspective is not seeing things very differently.

On the other hand, this bunch below is totally going to reinvent how you do everything. It may not be be pretty or efficient, but it’s going to be different.

Screen Shot 2018-10-21 at 15.57.59

Reinventing the world. Photo via Popular meme.

If your company and staff are great at industrialisation and six sigma improvements you need to be looking for opportunities to disrupt incumbent companies who have created a market where small improvements can be profitable due to volume. Nations sell themselves as great places for this sort of work.

Maybe your company takes good ideas that are just catching on and drives the move to a product with a great user experience and support. This is a different set of competencies.

New products and ideas must also come from somewhere. Big companies are not the place for having brand new ideas.

How to have a good idea

Wardley Maps show the climatic pattern of value moving right and becoming industrialised. But how do things appear on the left? Cynefin gives us a clue. We can potentially create ideas by removing constraints to move slightly into the chaos domain, and benefit from potentially different ways of doing things. A constraint you can remove is the need to immediately be profitable. Ideas from the Garage or Basement by a part-time entrepreneur fit here .

At research-based Universities, time and facilities for Academics to experiment in a safe to fail environment helps with new ideas. Cynefin requires at least one of the experiments to be naive. This happens when a chemist applies their heuristics to a physics problem. Cross disciple research groups provide this opportunity.

Universities have a whole raft of ways they exchange value with the environment. One of them is as an ideas factory. (I find the factory metaphor hilarious, so bad it’s good.)
This can be shown equally well with a Wardley Map or with Cynefin Dynamics. They show different perspectives and have different perspectives. Below is a Wardley Map with the needs of Academics to publish papers, and the needs of an external organisation whose conpetencies include marketing and manufacturing innovative products.


The need for immediate profit is one of the constraints that is removed to take advantage of ideas from chaos. New developments are either written up and published as papers, meeting the Academics need to produce visible output. If suitable the ideas may be patented before publishing. If the patent has an obvious application then companies requiring an innovative edge may licence the technology. The University spends the proceeds on more research and bureaucracy.

If the idea can’t be licensed, a Spin Out company may be created to develop a product up until the company can be sold. A University has no interest in developing and marketing a product, and the improvements in production required to meet pricing and all the other things required. They sell the company and get back to the area of the Wardley Map they are good at.

Putting it all together

This bog only shows only one pattern on the map. For more patterns and strategies see Simon Wardley’s book on medium.

Wardley Maps are a great visualisation that work well looking at how we add value personally with our skills, with the products our employer sells, and how our sector of industry is performing.

They form part of a set of visualisations looking at the entire value chain from what people do, all the way up. The world is getting disrupted at an astonishing rate, and mapping helps us to see what the future may hold.

Using maps we can discuss the sourcing of parts of the value chain, the approaches from agile to six sigma that may be used to deliver those parts. The culture of the organisation that are likely to excel in these domains can also be discussed. Finally, strategic patterns can be overlaid on the maps to discuss how to act in the future.

There are ideas in this post from Wardley Mapping, Cynefin, VSM, and loads of other approaches. Cynefin is a trademark of Cognitive Edge.

Scrum and VSM v1

Using the viable systems model understand and diagnose issues in Scrum and IT

System 5. Arrgh.

This article will look at Scrum through the lens of a Systems Thinking approach called the Viable Systems Model. Applied to organisation, the Viable Systems Model often feels like well thought out common sense, so the things VSM says may not be a surprise. What may be a surprise is the diagnostic power the VSM has both within and outside the Scrum team . This power makes the learning curve of the VSM worthwhile.

I’ll start looking at a Scrum team as a viable system and follow up this post with a discussion of the members of the Scrum team as viable systems. Finally in a 3rd post I’ll analyse various IT situations I’ve seen using the same tool. This should show the power of the VSM, and the range of situation it can be used in.

All this analysis is released under a Creative Commons License. It would be great to get feedback from both a VSM and Scrum perspective, more examples, and build up a resource.

Context, context, context

It’s worth noting here that application of Systems Thinking is always contextual. If you have a different context to me we’ll not have the same analysis. I’ll be treating this as a learning opportunity, rather than a quest for an uber-analysis that can can applied in any context. (hint: the Uber-analysis doesn’t exist).

I’m assuming you know Scrum and this article is an introduction to the Viable Systems Model, but it should work the other way. I’ll also show how the VSM can be used to diagnose issues with teams and IT in general, It’s a useful aid for Scrum Masters or Agile Coaches when things aren’t quite working. In fact, an Agile Coach, armed with the VSM and a couple of other people systems tools would be pretty formidable.

This power comes from the ability to name things that may be missing and we’d like to improve.

Having a name for things is vital if we want to talk to other people about them. The VSM helps us do this.


Ashby Law of Requisite Variety

The Viable Systems Model is Ashby’s Law of Requisite Variety applied to organization design. So I’ll really be looking at Scrum at IT issues using Ashby Law of Requisite Variety.

Rather than introducing the VSM academically (there are lots of academic papers that do that) I’ll use it to talk about Scrum, showing why the parts of Scrum work – and giving some general diagnostic advice when things are not working as expected.

Quick introduction to Viable Systems, and the One Weird Trick

Before I start there are a few thing about the Viable Systems Model that I’ll point out. Firstly the model is recursive. Viable systems contain viable systems. So viable teams make up viable organisation units, that make up viable organizations. Viable organization need to exist in viable sectors of the economy or society. A system exists within its environment, and Scrum teams are no different.

The other cool thing about the VSM is that the things we look for are at each recursion are exactly the same as every other recursion. This One Weird Trick makes the VSM easily applicable to lots of different organizations at different levels. The requirements are numbered System 1 to System 5, and there are purposeful communication channels between various parts of the system. There is also the systems environment that may have more than one element.

These parts of a viable system are called System1 (S1) to System5 (S5) , shown in the diagram below. System S3 gets an extra S3*, just because.



The VSM just says that the S1to S5 things need to exist. They can be done by the same person or separate people.

Applied to a Scrum team

Scrum, as described in the Scrum Guide is good example of a general way to organize to produce working software in complex situations.
Analysis of a Scrum Team as a Viable System shows what each part of Scrum is doing, and shows where there are assumptions about the sort of problem you are solving that may cause issues, and areas where other skills and approaches may be required.

The Scrum team and its customer

A Scrum teams customers are the people whose needs are met by the software that team writes. This can either be a group outside of the organisation like a small mobile game dev company that releases code straight to paying customers, or a team whose output needs to be integrated with the output of other teams in the organisation to provide value. This is important, but the implications come later.

The communication between the customer and the team is managed by the Product Owner or PO . The PO reduces the variety of requests coming from the customer, to something the team can cope with. In the other direction, the value from the team is increased by the Scrum Master, the Dev Team and the Product Owner putting together Sprint Goals that will meet needs in the best way.



Of course the developers can talk directly to the customer, but this is in the context of delivering value that has already been discussed and agreed.

Providing value VSM System 1

The parts of a Viable System that provide value to the customer/environment, is called the System 1. In a Scrum team there are 5 +/- 2 parts of this. The System 1’s need their output co-ordinating by a System 2.

The output of the team may need integrating into the output of other teams. These may be financial, marketing, physical product production, legal, regulatory or other Scrum teams. (oooh, maybe even other waterfall teams…) I’ll look at this with the VSM in another post, but for now, assume the Scrum Team provides value to the customer/environment directly.

Co-ordination VSM System 2

Co-ordinating system 1’s into a cohesive output is mainly the job of the Scrum Master. The SM makes sure that the daily standups, retrospecitves, and sprint planning all occur. The Scrum Masters job is also to remove any impediments to the team, reducing the variety of problems they need to deal with.
The team is also expected to self organise to deliver the Sprint Goal. Scrum mandates daily standups where the SM creates a space where it is safe to make mistakes and learn. This skills required to do this are outside the scope of the Scrum guide, but are vital to a well functioning team. System 2 is S2 in the triangle in the following diagram.

Getting resources and managing delivery -VSM Resource Bargain and Audit

VSM requires an organisation to have a resource bargain, where resources (Like a factory, employees and raw materials) are converted into something of value. The bargain (ie 1 factory, 20 people, raw materials) needs to be converted into something useful in a particular timeframe (ie 500 products in a week). This is a performance agreement that needs to be agreed and monitored. Scrums is designed to work in more complex environments that factory, so the resource bargain is more complex.



A Scrum team creates value with a team of developers, a Scrum Master, a product owner, IT resources and 2-4 weeks of time. This is a resource bargain with the organisation. The team is given resources in return for delivering value and meeting peoples needs. Because of the short sprint cycles, the resource bargain can be reassesed regularly to see if it is delivering what it promised. This is done at the sprint review, where the output of the sprint is delivered to customers.
The VSM says that the System 1’s need a in depth audit – that is carried out at irregular times. The regular Scrum Review is the closest to this audit, although it is not irregular, and it does not bypass the PO and SM. This means certain organisation anti-patterns may not be detected. In VSM this audit is called System 3*. Done without care this audit can break trust, so leaving it out of any model of the organisation can be disastrous.

Managing the Current Operations VSM System 3

This is where the whole of the the current operations are reviewed for effectiveness, so there is the output of all of the previous sprint goals, the current sprint goal, and future goals. There are decisions here made about the whole deliverable of value to the customer, including bug fixes, and any operational resources required.
In VSM this is the System 3. In Scrum the team develops new features. Scrum covers a some of this, but the output of the team may be passed to an operations / devops team. A lot of this is out of scope of Scrum. A Devops team would need to be a Viable System, so could be designed and diagnosed with VSM.

If the output of the Scrum team is integrated with the output of other teams to provide value, then the overall output will be assessed by it’s own System3. ie the performance of the whole product.

Thinking about the Future VSM System 4

The decision to continue developing the software, to stop developing software if progress is too slow, or it’s not providing value needs to be made. The team also needs to look ahead to see what the future may hold, and how the team will continue to be viable.

The Identity of the Team VSM System 5

The identity of the team is its purpose. A Scrum team would align to the values in the Agile Manifeto, and also to the Scrum Guide. Its purpose could be “meeting needs and delivering value using Scrum”. Agendashift  says  “We deliver <what> to >whom> so that <why it matters>”.

The activities it does (S1s) affects its identity more than what the system says its identity is. So if the team isn’t doing Scrum it isn’t a Scrum team. A team can’t just call itself Agile and use the right words. It need to behave in a way congruent with the agile manifesto , and this behaviour is supported by referring to the identity in System 5.

The identity of the team – the decision to use Scrum and be agile is owned both by the actions of the developers and people in the next System level up (if it exists), who have decided that Agile teams are required, and can integrate the work of the Scrum team into a product with value.

In following posts I’ll look at the next recursion up and down from the Scrum, looking at a developer, and the next level up. If you’re a SM or PO I’d be interested in modelling those roles too, but I’d need an expert in these domains to do this properly.

Links to further reading resources TBC

There are some in depth books by Stafford Beer and Patrick Hoverstadt covering the VSM.

I’ve personally found the VSM resources on the web to be generally quite poor.

Here are some good web resources.

Trevor Hilder into to VSM

Tom Graves Pervasives and the VSM algedonic link



Getting the Finger Pointing Right. How to fix tech problems under pressure.

These ideas and suggestions are from my experience with IT incidents solving problems with complicated and interdependent systems. In most cases accurate, timely and safe finger pointing was the key to getting the problem sorted. There are a few thing that need to be in place before this can happen. I’ll cover what I’ve learned fixing issues with colleagues.

I’ll cover

  • some ideas for solutions to the problems we have working together
  • issues with experts and non experts
  •  a framework for a team to solve problems under pressure
  • a way to share and collaborate across knowledge silos


IT problems with complicated Systems

Hard to fix problems are usually the ones no-one expected. Problems we expect have planned and tested responses. Often we design them out. The problems that are left have fell down the gaps in our shared knowledge.

The way we build things

The way we build things can make them hard to fix.

For example, if there are three specialist teams building your application, you’ll get an application with three parts. You may design it this way, or you may get a mirror of whatever your organization structure is. This Conway’s Law in Action.

Due to Conway’s Law, the architecture of your IT system may be decided by the division of work between teams, due to knowledge silos or desire for clarity of management structure, rather than optimizing running the finished system.

Solutions to problems are often where two knowledge silos meet and the silo locations were there before any code was written.

Hey! We need silos!

We can say that knowledge silos are bad. We need to get rid of them. The frustration with some forms of silo behaviour is understandable. For complicated systems, knowledge silos are inevitable. No one can know how everything works. Unfortunately Fixing Silos by bridging them, or breaking them down them creates more problems. The bridges and new structures become the new problem sites. It’s inevitable we’re going to have silos where deep technical expertise is required. The best we can do choose where these silos are, why they exist and how we manage the boundaries.

Who is in the room

Things have gone wrong, and people are in a room looking for a solution. Some are Subject Matter Experts, others may be responsible for teams involved, for communication or as customer advocates. There is likely to be senior management who is being asked by the CIO how long until it’s fixed.

To fix the problem quickly, the composition of people in the room is vital. You need the people who designed the system, configured it, wrote the tests and those who run it. There may be knowledge silos across teams here.

You also need someone who understands the big picture but not the details. This role is vital to ask fresh and simple question that experts may not see.

You also need someone to get extra resources you’ll need, book rooms, write communications, and document the decisions that were made, and approve changes that are made to mitigate and investigate the problem.

More Silo Dangers

In a high pressure situation, a lack of psychological safety makes people avoid suggesting diagnosis if they risk their opinion and their past work being judged.

Even where safety exists in a team, it may not exist between silos.
It may be career limiting in an organization to be seen to be the cause of an issue where learning is not valued.

Under pressure people gravitate to general, rather than technical skills (http://bit.ly/AllspawThesis), and under pressure we may defend our silo territory before looking to solve the issue.

Group Problem Solving Anti Pattern #1 Beginner > Expert

from @swardley

The more you know about something, the more you know you don’t know. Simon Wardley joked about this and seemed to hit on some visceral truth. There seems to be more than a shred of truth in this, as beginners are often more certain than experts.
People like to save face. Without work, cross silo meetings are not a safe spaces, being wrong is bad, being responsible for the system at fault may be worse.

Group Problem Solving Anti Patterns #2 – Hippos

Perhaps someone senior (the Hippo, Highest Paid Person in the rOom) has heard about X. That’s what they heard broke last time. They’d really like you check that right now. Maybe the person responsible for X puts forward a strong case why it definitely isn’t X. The defense works but the barriers are now up, and there is a round of finger pointing until someone risks losing an eye.
Worse, someone technical, knowledgeable and dilligent may reply “well I suppose it could be X” (see Anti Pattern 1). And a group goes off to look at X. (Pro tip : It’s probably not X).

Group Problem Solving Anti Pattern #3 Experts are also not enough

A Bongard Problem. What is the difference between all the images on the left, and all the images on the right? This sort of problem has all the information you need. You’re looking for patterns.

Experts are rarely enough if the incident was caused by something unexpected. Experts know what to expect and what they have seen before. The problems of being an expert in an open system, where something new has gone wrong has been covered in my post about Bongard Problems. These problems, where you have two sets of information – before and after – are very similar to IT problems. You’ve got two situations, and you’re looking for the difference that makes a difference. If you’re pattern matching things you’ve seen before, and you’re the expert so you’re not expected to be wrong, new problems can be hard to find.

Someone is required who can ask questions and doesn’t let what they already think they know get in the way. Someone who doesn’t have expert knowledge will find this easiest.

We need a facilitated solution

Someone in the room needs to step up and facilitate the group. It may be that everyone takes one step back and you end up being in charge.

Whoever facilitates need to make sure there is

An Incident framework to follow

What should we do, who should do it, what we do next. This is what goes in the ITIL service management box ‘fix the broken thing’. Ideally there is someone who books room, arranges test areas and gets any hardware or software required for the team fixing the problem.

A example P1 incident process to follow. This can be improved.

A overall End to End systems map

Very Simple Mind Map of a Client Server Application. The detail is hidden, and would go left to right. Top to bottom goes from the client to datacenter, and into the components.

This is not the architecture model, or a solution design. Those won’t do what we want.
The purpose of this map is to help solve problems, so it’s different from the documents produced in architecture and design. It’s owned by the DevOps or IT Operations & Development teams. It is a map of how all the bits fit together in a journey to get your users needs met. You can point at it, and everyone one in the room knows what your talking about. It allows fora shared mental model.

The key to the map, is that everyone agrees that it is correct, and it’s changed to reflect any new information with information being added to the map in the correct place. When people talk about the problem they can point to the map to show the area they are talking about. This can save a load of time and misunderstanding. It also mean that large areas can be ruled out, meaning that Anti Pattern #2 can be avoided.

We can then ask better questions, and make better decisions as we’re talking about the same thing.

We’re pointing at this map, not at each other! We ask questions about the map, everyone has a shared model of the system. We make better decisions.

This map was used to pinpoint a very hard to find problem that would occur for users about once every few day of constant use – and would “fix itself” after less than 5 mins.


Observe, Orient, Decide, Act

“What we do next” to find the solution can be answered by John Boyd. OODA, the US Military Engagement Strategy for complex fast changing situations is also great for approaching IT problems.

For other problem solving methodologies see Brendan Greggs excellent site.

OODA is a fast decision making loop. If you need to have an Emergency Change Approval Board before each Act, decide who’s on this and make it happen quickly if you want things fixed. This is a feedback loop that can run really quickly. Have the right people in place.


What do we know? What is instrumented? Is there information from users? How much do we trust the data? This information will have been collected and added to your map. Show what information we know where.


As an Incident Team, discuss what does this data mean? Where does it suggest the problem may be? What do we now know, and with what certainty?
You’ve probably just run and test, and you should already know what the results mean, but it’s likely you also have new data from other sources too.


Decide what test to run next. Everyone needs to agree what the potential results of the test means. Ideally the next test will rule out the largest part of the End to End map possible.

Using the End to End diagram, understand what the data may be suggesting. When evaluating proposed actions as “What is ruled out if the results are X, what is rules out if the result are Y”. Ideally you’ll design tests that give X or Y.

You need to decide as a Team what the results mean before the test is done. What the results mean should be visible on the End to End diagram. For example a test should allow you to rule out parts of the system.

Every test goes through this. So if the Highest Paid Person in the rOom thinks you should find what is being tested on the End to end diagram, and the things that can be learned from the results are discusses.


Run and document the test and results.

Where do the fingers point now?

Instead of pointing fingers at each other, the Incident Resolution Team are pointing fingers at the End to End Diagram, and asking questions like “what would reconfiguring this tell us” or “We could replace this component to rule out all of this part of the diagram”.

New ideas can be quickly evaluated, and extra detail is quick and easy to add where it is required.

We can ask better questions and make better decisions. When we are asked why, we can point at a map, rather than a person. That’s got to be better.

Clean Interviewing at Work

Photo by Alex on Unsplash

At work, I need to talk to people at work to understand their perspectives on how the organization functions. A large University is similar to a small town. We have lots of buildings and green space, a lake, car parks roads and buildings. And security guards. And lots of people trying to achieve their goals in different ways. We’ve campuses in the UK and Asia too, links to a large teaching hospital and our own Arts Theatre. We’re complex.

I’m collecting information for User Journeys for people working, studying teaching and researching. If we help these people reach their goals, the University reaches it’s goals too. We’re structurally coupled to their success. The success of students, staff, and faculty is the success of the University.

An example of a user journey is below. We can see how someone experiences traveling with Lego.

To collect information I’m talking to people using ideas from Clean Interviewing. My goal is to find out what people think using their own words. I’d like them to see the maps I build and recognize their journeys. By accurately reflecting their perspective I hope that they’ll advocate for the use of the maps.

In almost all aspects the conversations I have are normal. I’d not expect the person I’m talking to notice anything unusual. There really isn’t anything weird or unusual. Apart from the questions are usually of the following form, and use the exact words used. I don’t paraphrase.

A selection of the questions I use are:

  • What needs to happen here?
  • What is it called?
  • Is there anything else about <that>?
  • What happens before <that>?
  • Is there anything else that needs to be here, but isn’t?
  • What happens next?
  • Overall, is there anything missing?

What I’m doing is using a framework to develop a model of what things are, what they are called, and what needs to happen before and afterward. It’s a normal conversation, and every part doesn’t follow these rules. If there is something I’d need to find out, I’ll use a clean question.

These are really simple questions suitable for situations where I could assume that I knew what the words used meant, and what happened was the same as the other interviews I’d done. By removing the assumptions I believe I get better information.

Clean Interviewing is based on the work of David Grove, modeled by James Lawley and Penny Tomkins.  More details, including use in academic research settings, are on the Wikipedia entry. This blog post shows a very simple use of Clean Language Interviewing.

To learn more about Clean Interviewing there are events on https://cleanlearning.co.uk/events

Why would I care about Mikhail Bongard’s problems?

TL;DR It’s hard to solve problems that are open, experts won’t help, filter bubbles, why does it matter if you are angry reading all this, double loop learning Clean Feedback and anti-fragility.

Thanks to Chris Cupit, Tomasz Glazer and Richard Harris for the discussions around this topic, and to Andrew J. Baker for feedback and pointers.

What are Mikhail Bongards problems, and why should I even care?

Quick Overview

Mikhail Bongard invented a type of puzzle called a Bongard Problem. These problems are interesting because you get to see all the answers. You need to work out the question. Bongard problems have two sets of six images. The solution to the problem is the explanation of the difference between the two.

Concave vs Convex edges

These inductive reasoning problems help us understand how we think in the real world. The book Goedel Escher Bach provides a methodology for solving Bongard problems, with a slant towards using Artifical Intelligence. At first the AI community thought these problems would a straightforward path to implementing AI. They’re not. And they’re much simpler than the other two examples of real-world Bongard Problems I’ll use.

This post discusses how these problems can be seen in the real world, and how they are connected to learning, and our approach to life.

How to Solve Bongard Problems

An example of Hofstadter’s methodology from Goedel Esher Bach is below. Note that this methodology is wrong for the problem that is being solved. I’ll call Goedel Esher Bach GEB from now on.


The method involves having a vocabulary for the domain, and an understanding what Gregory Bateson calls “the difference that makes a difference”.

Ninety-Nine Problems

Mikhail Bongards book Pattern Matching has one hundred problems. Solving these we can keep adding to the “difference that makes a difference”. We can build a list of all the heuristics we need, becoming an expert in this set of Bongard Problems.

When this list exists we have a vocabulary that gives two different approaches available to someone new to the problems.

Approach One, an open problem

This first approach is when you don’t know what the potential rules are. For example, you don’t if there is a rule that “If the pictures were physical and upright, their center of gravity would make them unstable”.

If you don’t know what the potential rules are, the potential number of solutions is unlimited, and you’re going to invent some potential solutions that are not used. So you could make new never been seen Bongard problems from these guesses.

How may these problems be seen by an expert in the original 100 problems?

Approach Two, you know the rules

The second approach is solving the same set of problems, but you have the language of all the solutions. This doesn’t make the problems trivial, but it’s a different type of problem. You have expert knowledge in the domain, and you’re doing pattern matching. You can combine patterns, but you might not be looking for new ones.

The difference that makes a difference

I’ll use a distinction between knowing all the solutions in advance, and not know them in the following three examples. I’ll then add the extra case where you think you know all the answers, but there may be other answers. Or put another way, you have a consistent explanation for how the world around your problem works.

I’ll use three examples

  • Playing the tabletop game Zendo
  • Understanding Organizational Strategy
  • Investigating unlawful credit card harvesting from your website

Example 1: Zendo

Zendo is a board game based on Bongard problems, using inductive logic. It used 3 shapes in three colours, and the idea is that a particular configuration of coloured shapes is given on a card, and players need to create structures that follow the rule on the card, and then win by correctly guessing the rule. So the Bongard problem is a configuration of shapes that is correct, and one that is wrong. You get to offer new configurations and see if they match the pattern that is the difference that matters between the two shapes.

Example 2: Organizational Strategy

Consider the case when a company makes some strategic actions. What is their new strategy? How has it changed? There will be some visible action, so one half of your Bongard Problem is the organization running one strategy, the other is the organization with the current strategy.
If you can figure out their change of strategy you can react to it with your own. If you fail to understand their strategy or carry on regardless you may go out of business.

Examples 3: Computer Crime

In this article, David Gilbertson describes how he’s hidden some code to harvest credit card details in a website that the reader may either own or use. Many website use frameworks that have a load of code imported into the pages. The data theft is viable.

In this example one half of the Bongard problem is your unhacked site, the other is a potentially hacked version. The Bongard problem is to prove that there is a difference. i.e. You’ve been hacked.

Example 1: What happens when I play Zendo with the kids

When I first played Zendo with the family I chose not to look at the cards describing the patterns that we could look for, and I didn’t show the family. We started from scratch and didn’t have a vocabulary.

It was very frustrating to play, and we made some crazy guesses.

The vocabulary on the cards includes orientation, colour, number, and layout. Doorstop, cheesecake, and tower were in the solution vocabulary.

Cheesecake, Doorstop, and vertical orientation.

We could now attempt the easy questions. Having a vocabulary is a required part of solving the problem in GEB.

When my son decided to choose a medium difficulty question we started with the usual guesses, but when we didn’t seem to be getting close we started to get agitated. We didn’t know the rules of the problem domain. More precisely, we didn’t have the vocabulary for the potential patterns. The temptation to ask for a clue or to give up was huge, because this was a game, and games are supposed to be fun. We did, however, have loads of crazy guesses that were not on the game cards. Failure is frustrating, especially if you are not expecting it.

Zendo with work colleagues

I then played the game with the Systems Thinking group where I work. I didn’t give my colleagues any information about the vocabulary or patterns. They were quickly able to create new patterns that matched the rule but couldn’t describe the exact rule that was written on the card I was using.

“The yellow wedge is in the cheesecake orientation” said Francis

The problem they had was there were two potential orientations for one of the pieces but without knowing this it was not a difference they noticed. As the photo above shows, the wedge shape can be in the ‘cheesecake’ or ‘doorstop’ orientation. If we’d read the instructions this would have been obvious. If we don’t have a name for something we have a hard time spotting it. Even if it is the difference that makes a difference.

After this game, without looking at the potential Bongard problems Tomasz helpfully created a problem that wasn’t on any of the cards. He put together some clear blue and yellow pieces. His pattern rule was “structures that can look green”. He’d created a new type of problem, unconstrained by rules. (EDIT : Tomasz was unconstrained by ‘being an expert’)

Structures that look ‘green’, Tomasz Glazer.

How might an experienced player who is familiar with all the vocabulary and patterns on the cards go about solving this problem?

Example Two: Strategy

In business, our competitors’ strategies are unknown to us. They may be easy to spot, or they may confound us.(explained in Patterns of Strategy). There are examples of strategies on the web, that suggest a more complete list exists (here is a great list of Strategic Gameplay from Simon Wardley).

By Simon Wardley





I’m not familiar with all the strategies listed above, but I can recognize a few of them in the wild.

We are unlikely to get a clue from our competitors, but we can ‘give up’ and have a strategy of just carrying on what we’re doing, head in the sand. We think our strategy is fine.
We know there are more strategic possibilities out there. It’s not clear to me if there is a definitive list of strategies available that is considered complete.

Is a new strategy possible? I think it should be. Does knowing all previous strategies help? I’m not sure. Is it possible to think up new strategies that are not covered by known ones? I don’t know enough to know. Is strategy an open or closed problem?

Example Three: Fighting Computer Crime

After describing how he would insert code to harvest credit card details on websites, David Gilbertson goes through the ways that the reader, who is likely to be knowledgeable on computer security, would use to discover the exploit code. These ways are the patterns and vocabulary to solve the problem.

David then systematically shows how each of our ways of finding the code is flawed and sows enough doubt that we may not find his code.

We thought we had a full set of tools, but it’s clear we don’t. We can’t choose whether to learn our knowledge is flawed, it’s forced upon us. It’s a painful read, and it probably makes people angry enough to prove him wrong!

What’s just happened?

Thanks to Andrew J Barker (@ajbkr) from Tech Nottingham Slack for pointing me in this direction, but any mistakes here are mine.
What is happening when we face an open inductive logic puzzle and we don’t know the full range of answers? I’d suggest that we’re not just learning the answer, but we are double loop learning (see wikipedia, by Chris Aygryis). Our current problem-solving skills are not enough. We’re recognizing this and by getting some new tools.

Double loop learning is not just realizing there is something to learn, like a new computer language, or musical instrument, and learning it. Double loop learning is finding that your old tools don’t work on the problem you’re facing, maybe never really did. Your old thinking probably got you into the mess you’re in. You need to make a fundamental change to the way you think, and can’t go back.

There is often an unavoidable event that triggers this sort of learning. Maybe your business fails, and you see how it was obvious there were problems if only you’d seen them. Or credit card details are stolen from your website. You can’t say it was someone else’s fault. Maybe your partner leaves you unexpectedly, and once you see the reason you kick yourself for not seeing it.

Once you’ve learned something like this, it’s hard to even imagine what it would be like not to know. The past is another country.

I think this sort of unavoidable situation that can’t be explained by your current knowledge is often the catalyst for double loop learning. But our brains don’t like the cognitive dissonance. They would rather explain events any other way.

This is why we get angry when playing Bongard games when we don’t know the range of answers, and why many companies strategy is to keep doing the same thing.

Anti Fragile People

Some event that could break a fragile version of ‘us’, or be endured by a robust ‘us’ actually makes us stronger when we learn from it.

The event needs to be capable of hurting us – or making us angry enough to put up defences.

So what is stopping us getting over the double loop learning wall and being Anti Fragile? And what can help us over the wall?

Single loop learning feels so much nicer than double loop learning. Double loop Learning required we get past the cognitive bias that says we have a consistent explanation for how the bits of the world we care about works.

What’s stopping us?

We’re stopping ourselves.

What can help us?

Unavoidable feedback from hard failures can make us stronger. The feedback can also be endured or can break us. Exposure to the thinking that comes from hard failure makes us uncomfortable.

In many circumstances, feedback structured in a way that helps us develop, by making it hard for us to ignore. The best example of this is Clean Feedback, by Caitlin Walker.
That’s another post.

Really, what’s stopping us?

There is a useful (but ultimately wrong) model of how our brains work, called the Triune Brain. This suggests that there are 3 areas, or thinking states that we use. The lowest brain is the Reptilian brain. This is the fight, flight or freeze response. Knowing we are physically safe keeps this brain quiet. Feeling unsafe activates this brain. A fight also can mean get angry…

If our Reptilian Brain is happy we can use the Mammalian Brain. This brain has higher functions and is kept happy by knowing the rules. This can be social rules, like where the loos are, if you can just go and get a drink, and what time lunch is. Not knowing the rules puts you in the Mammalian brain.

If you’re safe, and you know the rules, then you can use the Neo-Cortex, or learning brain. I’m going to suggests that this is more accurately called the Single Loop Learning Brain. You can learn a new Language, or how to use your current skills better. It’s mostly saying “Don’t worry, you know the answers. No dissonance here.

To put all this together

Example 1: Zendo

We’re playing a game. It’s supposed to be fun. But if we only know a few of the possible answers. We don’t know the rules. We’re in the Mammalian brain. We’re not happy. It’s not terrible, but it’s not exactly fun either. We want to be in our Neo Cortex, getting the answers right. Being an expert.

Before we play again we look at all the cards with the answers on. The game is now to remember these and pick the right one. You can do this, we’re an expert. It’s now more fun, but we’ve limited our creativity.

Example 2: Strategy

With our full list of potential strategies, we’re working in our neo-cortex. An expert doing pattern matching.
Is a complete list a barrier to your brain double loop leaning? During analysis do we come up with new strategies that are likely to be wrong in the particular example, or do we match what we know?

A Strategy Expert, yesterday.

Are strategy consultants experts? How would they see a brand new strategy if it presented itself?

Example 3 Credit Card Harvesting

This article is delicious. David sets up the reader with a situation that’s hard to dismiss out of hand. It’s a challenge to the expert. He then uses the experts’ heuristics and shows how they are wrong. I imagine there may be a few angry readers.

Has this article about double loop learning offered any specific examples of double loop learning to the readers? Will it change anyone’s mind? Does it make people angry? Why is this article wrong?

So what?

To wrap all this up

  • There are open and closed problems
  • These can be played using Zendo, and the differences can be seen
  • Being an expert has some potentially negative consequences
  • Knowing the language to describe your problem could mean you miss things you don’t have a name for
  • Is this a difference described by being an engineer, and a scientist?
  • Unless using something like Clean Feedback, you need to be uncomfortable to experience double loop learning – learning that challenges your current way of doing things.
  • If you avoid discomfort you’re unlikely to learn how to act in the world better
  • What else can we do to learn about our understanding of the world?
    • A colleague suggested Drugs…
  • Talking to people we disagree with and understanding how they are cognitively consistent (or right…). I guess this can also be combined with the drugs 😉
  • What does this mean for our facebook and twitter filter bubble?
    • When someone makes us angry is it a chance to learn?
    • Or do we block and unfriend?

Knowing if the problem you’re solving is likely to be open or closed is a great starting point.

So lastly there are the Strategy problems I mentioned. Are they an open problem where new strategies are being thought up, or a closed one where they are all in a big book of experts strategy?

This may be a good direction to look in the future.