These ideas and suggestions are from my experience with IT incidents solving problems with complicated and interdependent systems. In most cases accurate, timely and safe finger pointing was the key to getting the problem sorted. There are a few thing that need to be in place before this can happen. I’ll cover what I’ve learned fixing issues with colleagues.

I’ll cover

• some ideas for solutions to the problems we have working together
• issues with experts and non experts
•  a framework for a team to solve problems under pressure
• a way to share and collaborate across knowledge silos

 

IT problems with complicated Systems

Hard to fix problems are usually the ones no-one expected. Problems we expect have planned and tested responses. Often we design them out. The problems that are left have fell down the gaps in our shared knowledge.

The way we build things

The way we build things can make them hard to fix.

For example, if there are three specialist teams building your application, you’ll get an application with three parts. You may design it this way, or you may get a mirror of whatever your organization structure is. This Conway’s Law in Action.

Due to Conway’s Law, the architecture of your IT system may be decided by the division of work between teams, due to knowledge silos or desire for clarity of management structure, rather than optimizing running the finished system.

Solutions to problems are often where two knowledge silos meet and the silo locations were there before any code was written.

Hey! We need silos!

We can say that knowledge silos are bad. We need to get rid of them. The frustration with some forms of silo behaviour is understandable. For complicated systems, knowledge silos are inevitable. No one can know how everything works. Unfortunately Fixing Silos by bridging them, or breaking them down them creates more problems. The bridges and new structures become the new problem sites. It’s inevitable we’re going to have silos where deep technical expertise is required. The best we can do choose where these silos are, why they exist and how we manage the boundaries.

Who is in the room

Things have gone wrong, and people are in a room looking for a solution. Some are Subject Matter Experts, others may be responsible for teams involved, for communication or as customer advocates. There is likely to be senior management who is being asked by the CIO how long until it’s fixed.

To fix the problem quickly, the composition of people in the room is vital. You need the people who designed the system, configured it, wrote the tests and those who run it. There may be knowledge silos across teams here.

You also need someone who understands the big picture but not the details. This role is vital to ask fresh and simple question that experts may not see.

You also need someone to get extra resources you’ll need, book rooms, write communications, and document the decisions that were made, and approve changes that are made to mitigate and investigate the problem.

More Silo Dangers

In a high pressure situation, a lack of psychological safety makes people avoid suggesting diagnosis if they risk their opinion and their past work being judged.

Even where safety exists in a team, it may not exist between silos.
It may be career limiting in an organization to be seen to be the cause of an issue where learning is not valued.

Under pressure people gravitate to general, rather than technical skills (http://bit.ly/AllspawThesis), and under pressure we may defend our silo territory before looking to solve the issue.

Group Problem Solving Anti Pattern #1 Beginner > Expert

The more you know about something, the more you know you don’t know. Simon Wardley joked about this and seemed to hit on some visceral truth. There seems to be more than a shred of truth in this, as beginners are often more certain than experts.
People like to save face. Without work, cross silo meetings are not a safe spaces, being wrong is bad, being responsible for the system at fault may be worse.

Group Problem Solving Anti Patterns #2 – Hippos

Perhaps someone senior (the Hippo, Highest Paid Person in the rOom) has heard about X. That’s what they heard broke last time. They’d really like you check that right now. Maybe the person responsible for X puts forward a strong case why it definitely isn’t X. The defense works but the barriers are now up, and there is a round of finger pointing until someone risks losing an eye.
Worse, someone technical, knowledgeable and dilligent may reply “well I suppose it could be X” (see Anti Pattern 1). And a group goes off to look at X. (Pro tip : It’s probably not X).

Group Problem Solving Anti Pattern #3 Experts are also not enough

Experts are rarely enough if the incident was caused by something unexpected. Experts know what to expect and what they have seen before. The problems of being an expert in an open system, where something new has gone wrong has been covered in my post about Bongard Problems. These problems, where you have two sets of information – before and after – are very similar to IT problems. You’ve got two situations, and you’re looking for the difference that makes a difference. If you’re pattern matching things you’ve seen before, and you’re the expert so you’re not expected to be wrong, new problems can be hard to find.

Someone is required who can ask questions and doesn’t let what they already think they know get in the way. Someone who doesn’t have expert knowledge will find this easiest.

We need a facilitated solution

Someone in the room needs to step up and facilitate the group. It may be that everyone takes one step back and you end up being in charge.

Whoever facilitates need to make sure there is

An Incident framework to follow

What should we do, who should do it, what we do next. This is what goes in the ITIL service management box ‘fix the broken thing’. Ideally there is someone who books room, arranges test areas and gets any hardware or software required for the team fixing the problem.

A overall End to End systems map

This is not the architecture model, or a solution design. Those won’t do what we want.
The purpose of this map is to help solve problems, so it’s different from the documents produced in architecture and design. It’s owned by the DevOps or IT Operations & Development teams. It is a map of how all the bits fit together in a journey to get your users needs met. You can point at it, and everyone one in the room knows what your talking about. It allows fora shared mental model.

The key to the map, is that everyone agrees that it is correct, and it’s changed to reflect any new information with information being added to the map in the correct place. When people talk about the problem they can point to the map to show the area they are talking about. This can save a load of time and misunderstanding. It also mean that large areas can be ruled out, meaning that Anti Pattern #2 can be avoided.

We can then ask better questions, and make better decisions as we’re talking about the same thing.

We’re pointing at this map, not at each other! We ask questions about the map, everyone has a shared model of the system. We make better decisions.

 

Observe, Orient, Decide, Act

“What we do next” to find the solution can be answered by John Boyd. OODA, the US Military Engagement Strategy for complex fast changing situations is also great for approaching IT problems.

For other problem solving methodologies see Brendan Greggs excellent site.

OODA is a fast decision making loop. If you need to have an Emergency Change Approval Board before each Act, decide who’s on this and make it happen quickly if you want things fixed. This is a feedback loop that can run really quickly. Have the right people in place.

Observe

What do we know? What is instrumented? Is there information from users? How much do we trust the data? This information will have been collected and added to your map. Show what information we know where.

Orient

As an Incident Team, discuss what does this data mean? Where does it suggest the problem may be? What do we now know, and with what certainty?
You’ve probably just run and test, and you should already know what the results mean, but it’s likely you also have new data from other sources too.

Decide

Decide what test to run next. Everyone needs to agree what the potential results of the test means. Ideally the next test will rule out the largest part of the End to End map possible.

Using the End to End diagram, understand what the data may be suggesting. When evaluating proposed actions as “What is ruled out if the results are X, what is rules out if the result are Y”. Ideally you’ll design tests that give X or Y.

You need to decide as a Team what the results mean before the test is done. What the results mean should be visible on the End to End diagram. For example a test should allow you to rule out parts of the system.

Every test goes through this. So if the Highest Paid Person in the rOom thinks you should find what is being tested on the End to end diagram, and the things that can be learned from the results are discusses.

Act

Run and document the test and results.

Where do the fingers point now?

Instead of pointing fingers at each other, the Incident Resolution Team are pointing fingers at the End to End Diagram, and asking questions like “what would reconfiguring this tell us” or “We could replace this component to rule out all of this part of the diagram”.

New ideas can be quickly evaluated, and extra detail is quick and easy to add where it is required.

We can ask better questions and make better decisions. When we are asked why, we can point at a map, rather than a person. That’s got to be better.

At work, I need to talk to people at work to understand their perspectives on how the organization functions. A large University is similar to a small town. We have lots of buildings and green space, a lake, car parks roads and buildings. And security guards. And lots of people trying to achieve their goals in different ways. We’ve campuses in the UK and Asia too, links to a large teaching hospital and our own Arts Theatre. We’re complex.

I’m collecting information for User Journeys for people working, studying teaching and researching. If we help these people reach their goals, the University reaches it’s goals too. We’re structurally coupled to their success. The success of students, staff, and faculty is the success of the University.

An example of a user journey is below. We can see how someone experiences traveling with Lego.

To collect information I’m talking to people using ideas from Clean Interviewing. My goal is to find out what people think using their own words. I’d like them to see the maps I build and recognize their journeys. By accurately reflecting their perspective I hope that they’ll advocate for the use of the maps.

In almost all aspects the conversations I have are normal. I’d not expect the person I’m talking to notice anything unusual. There really isn’t anything weird or unusual. Apart from the questions are usually of the following form, and use the exact words used. I don’t paraphrase.

A selection of the questions I use are:

• What needs to happen here?
• What is it called?
• Is there anything else about <that>?
• What happens before <that>?
• Is there anything else that needs to be here, but isn’t?
• What happens next?
• Overall, is there anything missing?

What I’m doing is using a framework to develop a model of what things are, what they are called, and what needs to happen before and afterward. It’s a normal conversation, and every part doesn’t follow these rules. If there is something I’d need to find out, I’ll use a clean question.

These are really simple questions suitable for situations where I could assume that I knew what the words used meant, and what happened was the same as the other interviews I’d done. By removing the assumptions I believe I get better information.

Clean Interviewing is based on the work of David Grove, modeled by James Lawley and Penny Tomkins.  More details, including use in academic research settings, are on the Wikipedia entry. This blog post shows a very simple use of Clean Language Interviewing.

To learn more about Clean Interviewing there are events on https://cleanlearning.co.uk/events

Overview

In the post  ‘Listen carefully, it’s the System talking I wrote about Barry Oshry’s Seeing Systems model. This describes the conditions we are in when working with other people, and how we can choose to behave in the relationship. I called these choices balcony or basement behaviours. Barry has an excellent book too.

I recently heard Caitlin Walker describe her method of Clean Scoping at the Metaphorum 2017 conference. This is an approach to understand or scope potential work to see if a Clean Language approach is suitable and is likely to work. The rest of this post discusses how I see these two approaches adding value to each other. I recommend Caitlin’s book ‘From Contempt to Curiosity‘ for more details.

---

Seeing Systems

In the Seeing Systems model, if we are trying to build a relationship with someone in the CUSTOMER condition, we’d like balcony customers, rather than basement customers. As someone responsible for the overall delivery of whatever a customer needs, we can choose to act as balcony TOPS.  A quick overview is:

Balcony TOP’s want to create a systems that can meet the challenges that they face. They empower people in the system to use their unique knowledge to improve the outcomes.

Balcony CUSTOMER’s engage in the details of what they need, provide feedback on the delivery progress, suitability and timing. Reading a bit more into Barry’s work I feel balcony CUSTOMER’s also see the power they have in using and developing the solution. They are not just asking for the answer provided to them.

Clean Scoping

Clean Scoping is part of Caitlin Walkers Clean Language and Systemic Modelling ™ approach, that i feel is a practical way of seeing if the necessary balcony conditions exist. In Caitlins case Clean Scoping is used to decide if she wants to work with the client or not. If we can’t choose our customers then we may try to influence them to behave in a BALCONY way.

Using the two models together allows us to understand what we are trying to do, and have a practical guide to having the conversations.

Caitlin is explicitly trying to create a system that is able to solve the problems it is trying to face. This is done by ensuring she is working at a sufficiently high level in the organisation to make sure the changes stick, ensuring that balcony customer behaviour exists, and transferring the skills to the customer so they are self sufficient.

Customer Behaviour

At my work organisation there is a group interested in how to develop and encourage balcony CUSTOMER behaviour from CUSTOMERs we work with. Catilin looks for this behaviour in potential clients at a high level in the organisation before agreeing to work. Described in her book, ‘From Contempt to Curiosity‘ Caitlin looks to encourage this behaviour – called Quadrant 3 behaviour – at different hierarchical levels of the organisation once there’s buy in. At my work we don’t get to choose our customers.

Using Clean Scoping questions, the organisational behaviour we want to have happen are balcony TOP, Balcony BOTTOM, balcony CUSTOMER.

Clean Scoping Questions

The help achieve this organisational behaviour, Clean Scoping questions would be:

• And what do we see and hear when <balcony behaviour>?
• When do you naturally get the <balcony behaviour> you’re hoping to get more of?
• What is happening at the moment?
• What is working well?
• What is not working well?
• What needs to happen, so what you would like to have happen is automatic?
• What would need to be true for people to naturally behave like this?
• What is happening at the moment?
  • Often Uncomfortable patterns are happening. This is often the difference between what we ask of others and what we do ourselves.
  • For example when we behave as a basement TOP with heirarchy, and expect others to behave as balconies. Behaviours are coupled.
  • Acknowledge what is true is true
    • Worldviews and perspectives are important here, and metaphor models can help
  • What would need to be be true for people to naturally behave like this? – People working to their strengths and acknowledging others strengths and contribution.

Biased and basement Behaviour

Behaviour from biases ensure that the patterns from the past continue. These are often confirmation biases that form part of the coupled relations in the Seeing Systems model. The blind reflex response is precisely why the relationships are here, and not in a better place. If we expect or behave with basement behaviour from another, we’ll get it in return – especially if there is organisational hierarchy.

Why and how

This post has covered some of the how questions for the why questions in the previous “It’s the system talking” post. There is a bit more to this…