Part 3 — What our relationship with risk reveals about leadership, judgement and accountability in the age of AI.
Are We Building Technology Around Challenges We've Never Fully Addressed?
We’re becoming increasingly sophisticated in how we develop, deploy and govern artificial intelligence, and the pace of that evolution shows little sign of slowing.
We have frameworks for managing AI risks, emerging standards for responsible use, governance models for autonomous systems and increasingly sophisticated tools for monitoring performance, security and compliance.
And we need them.
As AI capabilities evolve, so too do the risks associated with their design, deployment, use and management. Systems that once supported relatively contained tasks can now generate recommendations, influence decisions, coordinate activities and, increasingly, take actions on behalf of people.
These capabilities offer extraordinary opportunities. But they also introduce complexities that organisations cannot afford to overlook.
Throughout this blog series, I’ve been returning to a question that I believe deserves far greater attention.
Are we building increasingly sophisticated AI systems around leadership challenges we’ve never fully addressed?
Because while the technology is evolving, many of the human influences surrounding its implementation remain remarkably familiar.
The pressure to deliver results, the assumptions we make about what we know, the influence of personal experiences and beliefs, the decisions we make under competing priorities, and our willingness to question something that appears to be working.
These influences existed long before AI entered our organisations.
What concerns me isn’t simply that they remain present, but that technology can amplify their consequences, sometimes at a speed and scale we’ve never previously experienced.
And perhaps the greatest opportunity isn’t simply to develop better technology or stronger governance frameworks.
It’s to reconsider what we’re investing in before the technology is ever deployed.
What Happens Before the Outcome?
In Part 1, Are We Really Ready for Accountability in the Age of AI?, we explored responsibility, accountability and ownership, and questioned whether organisations genuinely understand how these dimensions operate beyond their formal governance arrangements.
In Part 2, When Accountability Becomes a Weapon, we examined how Clarity, Capability and Culture influence our willingness and ability to exercise accountability, including the roles of fear, organisational environments and Safe Zones.
But there’s another dimension to this conversation.
What if some of the most important opportunities to influence an AI-related outcome exist long before the technology produces a result?
Consider an organisation introducing an AI-assisted system to prioritise customer complaints and service requests.
The objective is reasonable: improve response times, allocate resources more effectively and provide a more consistent experience.
The system has undergone testing, performance measures have been established, and the organisation has invested in the necessary technical and governance arrangements.
Early results are encouraging.
But as the system becomes embedded in everyday operations, different individuals begin noticing different things.
One employee recognises that certain complaints involving vulnerable customers are receiving lower priority than their circumstances may warrant. Another, drawing on previous experience with similar technologies, sees the results as evidence that the system is working well. A third notices an opportunity to improve the process further but isn’t sure whether the organisation is interested in revisiting a solution that has already been approved.
None of these perspectives necessarily establishes that the technology is defective.
But each may offer information worth exploring.
And what happens next depends on more than the technology itself.
It depends on how those individuals interpret what they’re seeing, the experiences and assumptions they bring to the situation, the pressures influencing their decisions and the environment in which they’re expected to exercise judgement.
We often examine risk by looking at processes, systems, controls and outcomes.
But what happens when we look beneath the behaviour of the individuals interacting with those systems?
That’s where another perspective becomes particularly valuable.
S4R: Looking Beneath the Behaviour
In my book, Risk Starts and Ends with People, I introduced S4R (System4Risk) as a way of exploring the human influences that can contribute to risk to an individual and risk by an individual.
S4R begins with the individual. We first explore the risks to that person, considering the circumstances, influences and exposures that may affect them. From there, we examine the potential risks by that individual, recognising how their experiences, interpretations, decisions, actions and inactions may influence outcomes.
And because no two individuals are the same, exposure to similar circumstances doesn’t necessarily produce the same experience or response. What one person considers manageable may be deeply challenging to another, while something one individual barely notices may hold considerable significance for someone else.
Their experiences, beliefs, values, circumstances and responses can influence how they interpret situations, make decisions and behave. When we extend that individual understanding across a team, project, division or organisation, we begin to see a much richer picture of how human influences interact, where vulnerabilities may develop and what opportunities might otherwise remain hidden.
This is particularly valuable when examining risks associated with different roles, responsibilities, decisions and working environments, because people don’t necessarily experience or respond to the same circumstances in the same way.
I’ve explored S4R through different leadership and risk perspectives in earlier articles, including A Different Lens on Stewardship and Unearthing Leadership Lessons Through the S4R Lens. Here, I want to bring that thinking into our conversation about AI and accountability.
S4R considers four interconnected elements: Predisposition, Stressors, Triggers and Onset.
It’s not about categorising people, predicting behaviour with certainty or removing personal accountability. It’s about remaining curious about what may lie beneath the behaviour we observe, including influences that might otherwise go unrecognised.
And when we apply this lens to our AI-assisted customer service example, the picture becomes considerably more interesting.
Predisposition: What Are We Bringing Into the Situation?
We all bring experiences, beliefs, values, assumptions and learned behaviours into our decisions and interactions, regardless of our role or authority. Together, these influences help shape our individual ways of interpreting information, situations and the world around us.
Imagine two experienced professionals reviewing the same AI-generated recommendation.
One has spent years working in environments where data-driven decisions have consistently improved outcomes. Their experience has strengthened their confidence in analytical tools, and they may be inclined to trust the recommendation unless something clearly contradicts it.
The other has spent much of their career dealing with the consequences of flawed information, incomplete assumptions or decisions made without sufficient consideration of the people affected. Their experience may lead them to approach the same recommendation with greater scepticism.
Neither response is automatically right or wrong.
Both individuals bring valuable experiences and insights, but those experiences can also influence what they notice, what they overlook and the conclusions they’re inclined to reach.
And this is where predisposition becomes important.
It’s not simply about someone’s personality or attitude. It’s about understanding the individual influences that may shape how they interpret information, perceive risk, recognise opportunity and exercise judgement.
The risk isn’t necessarily that we see things differently. It’s that we may not recognise how our own predispositions influence what we’re seeing.
Stressors: What Else Is Influencing the Individual?
Stressors aren’t confined to the workplace.
We all experience pressures arising from our professional responsibilities, personal circumstances and the environments in which we live and work. Some are visible, such as demanding deadlines, competing priorities or increasing workloads. Others may be less apparent, including personal challenges, financial pressures or circumstances individuals understandably choose not to share.
And the significance of a stressor isn’t determined solely by how serious it appears to someone else.
Its influence depends, in part, on the individual experiencing it, their predispositions and the circumstances already affecting them.
In our AI-assisted service environment, someone may be under pressure to achieve performance targets. Another may be managing an increasing workload, while someone else may be experiencing circumstances outside work that affect their capacity to concentrate or respond.
These pressures don’t automatically lead to poor decisions, and we shouldn’t assume they will.
But they can influence how people interpret information, assess priorities and respond to situations that challenge their expectations.
And if we fail to appreciate that individuals operate within different circumstances, we risk assuming that the same environment will produce the same responses in everyone.
Triggers: When Something Changes
A trigger may be an event, interaction, situation or moment of realisation that activates a response within an individual.
Sometimes it’s connected to a deeply held value, belief, sense of purpose or previous experience. Something that appears relatively insignificant to one person may have profound meaning to another.
And the change can be extraordinary.
One moment, someone may appear comfortable and engaged. The next, something is said or happens that fundamentally changes how they feel, think or interpret the situation.
It’s almost like a switch being flicked. The internal intensity can rise sharply, even when nothing appears to have changed on the surface.
But not every trigger produces the same response, nor does every response lead to a significant change in behaviour.
Some reactions diminish with time. Others may contribute to a more profound shift in how someone understands themselves, others or the circumstances surrounding them.
And triggers aren’t always associated with fear, anger or distress.
Sometimes they’re moments of recognition, when an individual suddenly connects information, identifies a discrepancy or realises something they hadn’t previously understood.
Returning to our AI example, imagine the employee who has noticed the treatment of certain customer complaints.
Perhaps they discover that the system is consistently overlooking circumstances that conflict with their understanding of fair treatment. That realisation may strengthen their determination to investigate.
For another individual, the same information may challenge their confidence in the system or their assumptions about how decisions are being made.
What matters isn’t simply the event itself, but what it activates within the individual.
Onset: When the Individual's Behaviour Changes
Onset is where we begin to recognise a meaningful shift in an individual’s behaviour, actions or inactions, influenced by their predispositions, stressors and triggers.
Sometimes that change is immediate and obvious. At other times, it may be subtle, concealed or only recognised much later.
And it isn’t necessarily negative.
Someone may become more determined to challenge a decision, pursue an opportunity or change their direction. Another may withdraw, stop contributing ideas or quietly reconsider their commitment to something they previously supported.
The significance of these changes cannot always be understood by comparing one person’s behaviour with another’s.
We need to consider the individual, their predisposition, their established patterns and the circumstances influencing them.
A person who has always been forthcoming may suddenly become silent. Someone who has consistently supported an initiative may begin distancing themselves from it. Another who has previously avoided confrontation may become unusually determined to challenge an assumption.
To someone unfamiliar with the individual, these changes might appear insignificant or entirely ordinary.
But considered against their previous behaviour, they may indicate that something significant has changed.
Sometimes the most significant change isn’t what someone begins doing. It’s what they no longer do.
Of course, a change in behaviour doesn’t automatically indicate a problem, and it certainly doesn’t justify assumptions about someone’s intentions. People change for many reasons, and understanding those reasons requires curiosity, respect and an appreciation of individual circumstances.
This is why S4R can be so valuable. It encourages us to better understand individuals and their environments, look beyond the immediate event and consider how risk may have been developing, sometimes long before its consequences become visible.
And when we apply that understanding across the individuals involved in a process, project or organisation, we gain opportunities to recognise influences and interactions that a conventional examination of systems and controls may not reveal.
What AI Can Amplify
Now consider what happens when an AI capability is no longer simply making recommendations but is authorised to take actions within defined boundaries.
An AI agent might process transactions, update records, initiate communications or coordinate activities across multiple systems.
These capabilities introduce genuine technical and governance challenges involving access, permissions, reliability, security, monitoring and delegated authority. They require appropriate technical safeguards, testing, assurance and oversight.
But they also raise familiar human questions.
Who decided what authority the system should have? What assumptions informed that decision? Whose knowledge was sought, and what perspectives were overlooked? How were concerns addressed, and who has the authority and willingness to intervene when circumstances change?
I was reminded of these questions while reading Mark Bernard’s discussion, Who Controls Your AI Agents?, which explored governance across the lifecycle of increasingly autonomous AI capabilities.
One exchange in the comments particularly caught my attention. Ryan raised a familiar fraud scenario: an invoice arrives containing instructions to redirect payment to a different bank account. His observation highlighted the danger of introducing an AI-enabled assistant into that process without appropriate safeguards. We could end up with invoice redirection fraud carried out by a faster clerk.
The concern isn’t simply whether an AI agent can read an invoice, extract information or update a payment record. It’s whether the organisation has established appropriate boundaries around what the agent is authorised to do, what must be independently verified and when human approval is required.
After all, changing payment details shouldn’t become an authorised action simply because a person, or an AI system, encountered an instruction in a document.
Invoice redirection fraud isn’t new. Neither are the risks associated with inadequate verification, misplaced trust or poorly defined authority. But when we introduce increasingly autonomous technology into a process without adequately examining those vulnerabilities, we may accelerate the very weaknesses we should have addressed.
And if the people responsible for designing, authorising and overseeing that process are operating with assumptions that haven’t been challenged, increasingly sophisticated technology won’t necessarily correct them.
Automation can increase our capacity to act. It doesn’t automatically improve the judgement behind what we authorise.
And perhaps this is where the AiLlusion becomes particularly dangerous: when the sophistication of our technology creates an impression of organisational maturity that the underlying leadership, decision-making and accountability may not support.
That’s not an argument against AI. It’s a reminder that technological sophistication and leadership maturity are not interchangeable. And when we confuse the two, we risk accelerating weaknesses that were present long before the technology arrived.
Which raises another question.
If we understand that technology can amplify existing vulnerabilities, why aren’t we investing more deliberately in recognising and addressing those vulnerabilities before they become embedded in the systems we’re building?
The Investment We Make Before the Risk Materialises
For many years, I’ve encouraged organisations to consider the difference between investing upfront to understand and address risk, and paying for the consequences of risks that were never adequately explored.
It’s a distinction that becomes even more important as organisations embrace increasingly autonomous technologies.
When we invest in clarity, capability, constructive challenge and environments where people feel able to share what they know, we create opportunities to recognise weaknesses earlier, improve testing, strengthen safeguards and make better-informed decisions.
We also create opportunities to identify possibilities that might otherwise remain hidden.
Think again about our AI-assisted customer service system.
If the organisation genuinely welcomes different perspectives, the employee who notices an unusual pattern has an opportunity to contribute valuable information before the issue becomes widespread.
The individual who sees potential improvements can help strengthen the process rather than simply accepting the existing design. And those responsible for the technology can explore whether its assumptions, testing and controls adequately reflect the realities of the work.
None of this guarantees that every concern will be identified, every decision will be correct or every adverse outcome will be prevented.
But it changes the opportunities available to us, giving us a longer runway to recognise concerns, explore possibilities and influence outcomes before our options become more limited.
Rather than discovering weaknesses only after they have become significant problems, we may recognise them while there is still time to influence the outcome.
And when something does go wrong, an organisation that has invested in these foundations may be better equipped to recognise the issue, respond constructively and limit the consequences.
The investment we make before a risk materialises can influence how far its consequences reach long after the initial event.
That’s something I often refer to as reducing the risk tail: the continuing exposure, disruption, financial cost, reputational damage and other consequences that can follow an adverse event.
It’s not simply about reducing the likelihood of something going wrong. It’s also about strengthening our ability to respond, recover and learn when it does.
And that investment isn’t limited to policies, systems or controls. It includes the people who make decisions, recognise concerns, challenge assumptions and influence how those systems are designed and used.
Because when organisations invest in the right foundations, they aren’t simply protecting themselves from potential losses. They’re creating the conditions for better decisions, stronger collaboration, greater innovation and opportunities that may otherwise never be recognised.
Risk isn’t only about what we might lose. It’s also about what we might never discover if we’re unwilling to look beyond what we already know.
These benefits extend beyond the immediate project or technology. Over time, they can strengthen organisational resilience by improving how people recognise emerging challenges, adapt to changing circumstances, respond to disruption and recover when things don’t go as planned.
Accountability Begins Long Before Something Goes Wrong
And this brings us back to where our conversation began.
We started by questioning whether organisations genuinely understand responsibility, accountability and ownership, and how these dimensions operate when increasingly sophisticated technology enters the equation.
We’ve since explored the human influences that shape accountability, the environments in which people exercise judgement and the value of looking beneath behaviour to understand how risk may be developing.
None of these perspectives replaces technical governance, appropriate controls or the legal and ethical obligations associated with AI. They help us examine the human dimensions surrounding them and strengthen how those obligations are understood and exercised.
Because a governance framework may define authority, but it cannot guarantee sound judgement. A policy may require concerns to be escalated, but it cannot guarantee that people will feel willing or able to raise them. And an AI system may operate within its authorised boundaries while still producing consequences that expose weaknesses in the decisions surrounding its design and use.
We can continue investing in increasingly sophisticated technology, stronger controls and more detailed governance frameworks. And we should.
But if we’re not making a comparable investment in the people, judgement and environments surrounding those systems, we may be strengthening one part of the equation while leaving another exposed.
The real opportunity isn’t simply to determine who will be accountable when something goes wrong. It’s to exercise accountability early enough to influence what happens next.
To create environments where people are willing to challenge assumptions, share concerns, explore possibilities and take responsibility for their decisions. To recognise that the risk we address today may be the consequence we never have to manage tomorrow.
And to understand that the maturity of our leadership will influence whether AI becomes a source of sustainable opportunity or an amplifier of vulnerabilities we should have recognised much earlier.
Perhaps the greatest challenge isn’t how quickly AI is evolving, but whether our leadership judgement, courage and accountability are evolving with it.
Because AI may be accelerating the future, but leadership still determines much of what we carry into it.
And the questions we should be asking aren’t simply about what our technology can do. They’re about what we’re prepared to understand, challenge and take responsibility for before we ask it to do more.
If we know that the decisions we make before deployment can influence the consequences we face afterwards, why are we still so willing to invest in technology without making an equally deliberate investment in the people and environments surrounding it?
Because risk starts and ends with people.
And sometimes, the most important place to begin is with ourselves.
Risk Rebels… What say you?


