Measurement, Risk, Legacy, and Change
By the end of this module you will be able to:
- Apply the strangler fig pattern and anti-corruption layer to a legacy migration scenario
- Quantify technical debt and prioritise remediation using business risk criteria
- Build a benefits realisation management plan that links digital programme outcomes to named owners and measurement periods
Real-world case · HMRC IT programme, 2004 to 2011
£10.4 billion spent. 34% of the intended value never realised. Not because the systems failed.
HM Revenue and Customs (HMRC) spent £10.4 billion on IT between 2004 and 2011. A National Audit Office review in 2011 found that 34% of the intended value was never realised. Not because the systems did not work, but because the benefits were never measured, the responsible owners were never named, and the organisational changes required to capture the benefits were never made.
This is not an unusual finding. The Standish Group's CHAOS Report 2020 found that only 16% of IT projects are delivered on time, on budget, and to specification. McKinsey's 2018 research found that only 30% of digital transformations achieved their stated goals. In both cases, the primary failure mode is not technology. It is the gap between delivering a working system and realising the value the system was intended to create.
Benefits realisation management is the discipline that closes this gap. It is the final module of the Digitalisation course, and it examines why technology delivery is the easier half of the problem.
The systems worked. So why did a third of the intended value disappear? What would it have taken to capture it?
With the learning outcomes established, this module begins by examining digital transformation roi measurement in depth.
15.1 Digital transformation ROI measurement
Benefits realisation management is the practice of identifying, planning, measuring, and actively managing the realisation of benefits from a programme or project. It ensures that intended value is defined before delivery, measured during and after delivery, and attributed to named owners. Without it, programmes declare victory on technology delivery while the value case remains unproved.
The Balanced Scorecard (Kaplan and Norton, 1992) provides a four-dimension framework for measuring organisational performance: financial, customer, internal process, and learning and growth. Applied to digital transformation, it produces a measurement framework that captures value across dimensions rather than reducing everything to financial return.
The financial dimension for digital programmes includes cost per transaction (digital vs legacy), revenue attributable to new digital channels, infrastructure cost reduction from cloud migration, and reduction in reconciliation and error-correction costs.
The customer dimension includes digital channel NPS (Net Promoter Score), task completion rate for digital services, customer effort score for key journeys, and abandonment rate at key funnel stages.
The internal process dimension includes deployment frequency (the DORA metrics from the Applied stage), mean time to restore (MTTR), automated test coverage, and time to market for new features.
The learning and growth dimension includes digital skills coverage (the percentage of the workforce with defined digital competencies), OKR achievement rate, and internal mobility into digital roles.
“The GOV.UK Performance Platform required every central government service to publish four standard metrics: cost per transaction, user satisfaction, digital take-up, and completion rate. Services that improved digital take-up while completion rate declined were not considered successful.”
Government Digital Service, GOV.UK Performance Platform, 2013-2019 - Measurement framework and service standard
This is a practical application of a multi-dimensional measurement framework at public scale. The four metrics together prevent optimising one dimension at the expense of another. A service that shifts users online but fails to help them complete their task has not improved; it has moved failure from one channel to another.
Measuring ROI requires knowing what value looks like. Before that value can be realised, legacy constraints must be understood and quantified. Section 15.2 covers technical debt - how to measure it and how to prioritise what to address first.
15.2 Technical debt quantification
Technical debt is the accumulated cost of deferred technical decisions: code written quickly rather than well, systems not updated when dependencies changed, architecture compromises made to meet deadlines. Technical debt is not inherently bad. Like financial debt, it may be rational to take it on in the short term. It becomes a problem when it accumulates unmanaged.
Technical debt creates compounding costs. A codebase with high debt is slower and riskier to change. Features take longer to build. Bugs are harder to find. New developers take longer to become productive. Ward Cunningham, who coined the term in 1992, emphasised that the cost of technical debt is the "interest" paid on every subsequent change made to the indebted system.
SQALE (Software Quality Assessment based on Lifecycle Expectations) is a method for calculating the estimated remediation effort for each debt item, expressed in time. A SQALE analysis of a codebase produces a debt ratio: the remediation effort as a percentage of the effort required to build the system from scratch. A debt ratio above 5% is generally considered high risk.
Business risk scoring adjusts technical debt prioritisation by the business impact of the affected system. A debt item in a payment processing system is higher priority than the same debt item in a rarely used reporting tool, because the business consequence of failure differs. Technical debt metrics without business risk weighting produce remediation lists that prioritise the cleanest code rewrites rather than the highest-risk debt.
Common misconception
“Technical debt should be eliminated before a programme delivers new features.”
Technical debt management is about prioritisation, not elimination. Eliminating all technical debt before delivering features would halt delivery indefinitely - every codebase accumulates some debt. The effective approach is to quantify debt by business risk, set a target debt ratio per quarter, and address the highest-risk items first. Ward Cunningham, who coined the term, emphasised that some technical debt is rational: the interest (future change cost) is worth paying if the principal (the deferred decision) enables timely delivery. The target is managed debt, not zero debt.
Technical debt quantifies the cost of deferred decisions within a codebase. Legacy system risk is the broader question of whether a whole system should be maintained, modernised, or replaced. Section 15.3 covers a structured five-dimension assessment for making that decision.
15.3 Legacy system risk assessment
A legacy system is a system that is outdated in some technically significant way: unsupported software or hardware, an obsolete programming language, missing documentation, unavailable original developers, or tightly coupled architecture that prevents independent change. Legacy status is contextual, not merely age-related.
Not every old system is legacy in the risk sense. A 20-year-old system that is well-documented, runs on supported infrastructure, and has an active maintenance team may be lower risk than a 3-year-old system built by contractors who have since left. A legacy risk assessment should evaluate five dimensions:
- Supportability - are the operating system, runtime, and language versions still receiving security updates?
- Maintainability - can changes be made safely, with test coverage and documentation?
- Dependencies - does the system rely on third-party integrations or hardware being discontinued?
- Exposedness - what is the business impact if this system is unavailable for one hour, one day, one week?
- Replaceability - what are the realistic options for replacing or modernising this system and at what cost?
Common misconception
“Legacy systems should be replaced as quickly as possible because they are inherently risky.”
Legacy risk is contextual, not inherent. A well-maintained 20-year-old system may be significantly lower risk than a 3-year-old system built by contractors who have left and left no documentation. The appropriate response to a legacy system depends on its specific risk profile: supportability, maintainability, exposedness, and replaceability. Some legacy systems are worth maintaining carefully for years; others require urgent replacement. Risk assessment before investment is how you tell the difference.
Once a legacy system is assessed as a candidate for replacement, the question becomes how to replace it safely. Section 15.4 covers the main migration patterns and when each one is appropriate.
15.4 Legacy migration patterns
Several patterns exist for migrating from legacy systems to modern replacements. The choice depends on risk tolerance, available funding, and the degree to which the legacy system can be isolated.
Big bang migration replaces the entire system at once, over a defined cutover window. TSB Bank's 2018 migration is the canonical UK example of big bang migration risk. A technical migration that locked out 1.9 million customers for weeks, triggered a regulatory investigation, and cost TSB approximately £330 million in remediation. Big bang migration is generally appropriate only for systems with no feasible incremental path or very low business criticality.
The strangler fig pattern is named by Martin Fowler after the strangler fig tree, which gradually envelops and replaces its host tree while the host continues to live. A proxy layer intercepts requests to the legacy system and routes them to either the legacy system (initially) or the new replacement (progressively, as capabilities are built). The legacy system is not switched off until all functionality has been replaced. At every point in the migration, functionality that has not yet been migrated continues to work through the legacy path. Defects in the new system affect only the migrated subset, not the entire service.
The anti-corruption layer is a translation layer between the new system and the legacy system that converts the legacy system's data models and interfaces into the new system's domain model. The anti-corruption layer prevents the new system from being designed around the constraints of the legacy system. Without it, new systems frequently inherit the complexity and limitations of the systems they are replacing.
“HMRC's Making Tax Digital programme built new digital filing capabilities as separate services, exposed via APIs, while the legacy core system continued to hold the tax record. Each new digital capability replaced a legacy interaction incrementally.”
HMRC, Making Tax Digital programme, 2019 onwards - Programme architecture: strangler fig implementation
The API layer acted as a proxy. The new digital services consumed legacy data via the API without knowing the underlying COBOL system structures. This is the strangler fig pattern in practice: progressive replacement with the legacy system continuing to function throughout. Each migrated capability reduced reliance on the legacy system while new capabilities were built.
The strangler fig pattern's additional time and cost compared to big bang are an investment in resilience. If a migrated capability has defects, they affect only that capability, not the entire service. The programme can pause, fix, and continue. With big bang migration, a defect at cutover affects all users simultaneously, with no operational fallback.
Migration patterns manage technical risk. Programmes must also manage execution risk - the forces that cause IT projects to miss scope, schedule, and budget. Section 15.5 covers the research on what actually predicts digital programme failure.
15.5 Digital programme risk
The CHAOS Report, published annually by the Standish Group, tracks IT project outcomes. The 2020 report found that 31% of IT projects were cancelled before completion, 53% were challenged (over budget, over time, or under scope), and only 16% were delivered on time, on budget, and to specification. These figures have improved modestly since the Standish Group began reporting in 1994, but they remain deeply unflattering to the industry.
The CHAOS Report's analysis of success factors consistently identifies scope management, executive sponsorship, user involvement, and iterative delivery as the top predictors of success. Notably, technology choice does not appear in the top predictors. The quality of project governance and the degree of stakeholder engagement matter more than the specific tools used.
SAFe (Scaled Agile Framework, version 6.0) is widely used in large organisations to coordinate agile delivery across multiple teams. SAFe's portfolio management practices include PI (Programme Increment) planning (quarterly face-to-face or virtual planning events involving all teams on a programme), continuous programme execution, and relentless improvement retrospectives. SAFe provides the coordination scaffolding for programmes that cannot operate as a single agile team.
Programme governance addresses structural failure modes. Organisational resistance is the human failure mode that governance alone cannot prevent. Section 15.6 covers the four patterns of change resistance in digital programmes and how to address each.
15.6 Organisational change resistance
Change resistance is normal. Prosci's ADKAR model (Awareness, Desire, Knowledge, Ability, Reinforcement) provides a diagnostic framework for understanding where in the change adoption process individuals or teams are stuck. An ADKAR analysis identifies the specific barrier rather than treating all resistance as the same problem.
Common patterns of change resistance in digital programmes include awareness gaps (staff who do not understand why the digital change is happening and assume it threatens their role), skills anxiety (staff who understand what is changing but lack confidence in their ability to work in the new way), process ambiguity (new digital processes designed for the technology but not for the people operating them, with unclear escalation paths and undefined edge cases), and incentive misalignment (staff measured and rewarded on metrics from the old operating model who resist changes that improve customer outcomes but reduce their apparent performance).
Incentive misalignment is the most structurally stubborn form of resistance because it cannot be addressed through communication or training alone. A contact centre team measured on call handling time will resist digital self-service tools that reduce call volume, their productivity metric, even if the digital tools are better for customers. Resolving incentive misalignment requires changing the measurement system, not the communication strategy.
Change resistance describes why organisations fail to realise intended value. Benefits realisation management is the discipline that makes value realisation a governed, measurable commitment rather than an aspiration. Section 15.7 covers how to build a benefits register that survives programme closure.
15.7 Benefits realisation management
Benefits realisation management converts a programme's intended value into a structured plan with owners, measures, and review dates. A benefits register must include six elements for each benefit:
- Description - what will change and for whom, expressed in terms of the user or stakeholder who experiences the improvement
- Measurement method - the specific metric, data source, baseline value, and target value
- Named individual owner - not "the programme team"; a named individual who will exist after programme closure and can be held accountable
- Realisation date - when the benefit will first be measurable and when it is expected to reach its target level
- Explicit dependencies - the operational changes (process redesign, staff retraining, manual process decommission) that must occur alongside technology delivery for the benefit to materialise
- Progress status - updated at defined intervals throughout and after programme delivery
NHS 111 Online was introduced in 2017 to provide symptom assessment and clinical triage via a digital service, reducing demand on the NHS 111 telephone service. The benefits case included a cost per contact reduction (£2 for digital vs £18 for telephone), reduced call waiting times for telephone users, and improved clinical safety through standardised assessment. The benefits realisation plan assigned ownership to NHS England's urgent care directorate, specified baselines from 2017 call data, and included quarterly measurement reviews. By 2022, NHS 111 Online was handling over 1 million contacts per month.
“MSP defines benefits management as the identification, definition, planning, tracking and realisation of benefits throughout the programme lifecycle and beyond.”
Managing Successful Programmes (MSP) 5th edition, Axelos, 2020 - Chapter 4: Benefits management
MSP 5th edition provides a benefits management strategy template, a benefits map linking outputs to outcomes to benefits, and a benefits realisation review process. The key word is 'beyond': benefits realisation continues after the programme closes, which is why named owners and measurement plans must survive the programme structure.
Technology delivery is approximately 20% of the problem. Organisational change, capability building, and sustained leadership attention are the remaining 80%. A programme that delivers its technical scope on time but fails to achieve the resulting operational change has delivered technology, not transformation.
A large UK utility company is migrating 4 million customer accounts from a 15-year-old billing system to a modern cloud platform. The migration team proposes a big bang migration over a single weekend. The programme director endorses this because it avoids running two systems in parallel. A risk manager recommends the strangler fig pattern instead. The programme director argues the strangler fig will take longer and cost more. Which response best addresses the programme director's argument?
A regional council has delivered a digital transformation programme covering 12 services, on time and within budget. Six months later, an internal audit finds that cost per transaction for eight of the twelve services has not changed: residents are using the new digital services but staff are still completing parallel manual processes. Which failure in programme design most directly caused this outcome?
A digital programme manager is told that 'the development team has paid down all the technical debt'. A SQALE analysis of the codebase shows a 2% debt ratio, but the analysis was done without business risk weighting. A risk manager is concerned. Which scenario best illustrates why the risk manager's concern is valid?
A programme manager is creating a benefits register for a new digital HR system. They include: 'Benefit: reduced HR administration time. Owner: HR programme team. Target: 20% reduction. Date: six months after go-live.' An MSP practitioner reviewing the register flags three problems. Which three are most likely to be identified?
Key takeaways
- Digital transformation ROI measurement requires a multi-dimensional Balanced Scorecard approach: financial, customer, internal process, and learning/growth dimensions each capture value that the others miss.
- Technical debt is manageable when quantified and weighted by business risk. Remediation lists prioritised by code cleanliness rather than business impact consistently address the wrong problems.
- The strangler fig pattern enables legacy migration with a controlled fallback at each step. Big bang migrations are appropriate only for low-criticality systems with no viable incremental path.
- The CHAOS Report (Standish Group 2020) finds only 16% of IT projects fully succeed. The primary predictors of success are scope management, executive sponsorship, user involvement, and iterative delivery, not technology choice.
- Benefits realisation management requires named individual owners, specific measurement plans with baselines and targets, and explicit dependencies between technology delivery and operational change. Programmes without these elements deliver technology but not transformation.
Standards and sources cited in this module
Standish Group, CHAOS Report 2020
IT project success, challenged, and cancelled rates
Provides the project outcome statistics cited in Section 15.5 (31% cancelled, 53% challenged, 16% fully successful). The success factor analysis is cited to establish that governance and stakeholder engagement matter more than technology choice.
Fowler, M., StranglerFigApplication, martinfowler.com, 2004
Pattern description and rationale
Defines the strangler fig migration pattern analysed in Section 15.4. The HMRC Making Tax Digital example in the same section is drawn from the pattern's application in the UK public sector.
Chapter 4: Benefits management strategy, benefits map, and realisation review
Provides the benefits realisation management framework quoted in Section 15.7 and applied in the quiz. The benefits register requirements (named owner, measurement method, dependencies) are drawn from the MSP benefits management practice.
Kaplan, R.S. and Norton, D.P., The Balanced Scorecard, Harvard Business Review, 1992
Scorecard perspectives: financial, customer, internal process, learning and growth
Provides the balanced scorecard framework applied to digital transformation ROI measurement in Section 15.1. The GOV.UK Performance Platform example illustrates the same principles at public sector scale.
SAFe 6.0, Scaled Agile Inc., 2023
Portfolio management: PI planning, programme execution
Cited in Section 15.5 as the coordination framework for large digital programmes. PI planning and the SAFe programme increment structure are referenced as mechanisms for managing multi-team programme risk.
NAO, Managing the HMRC IT estate, 2011
Value realisation findings
Provides the opening case study: £10.4 billion spent, 34% of intended value unrealised. The NAO finding establishes the stakes of benefits realisation failure at programme scale.
Prosci, ADKAR model for change management
Awareness, Desire, Knowledge, Ability, Reinforcement
Provides the change resistance diagnostic framework in Section 15.6. The four patterns of resistance (awareness gap, skills anxiety, process ambiguity, incentive misalignment) are mapped to ADKAR stages.
This completes the digitalisation course. You now have a structured framework spanning definitions, drivers, building blocks, data standards, platforms, pipelines, measurement, integration, architecture, operations, strategy, trust, ecosystems, and change management. Return to the course overview or explore the summary and games section to consolidate your learning.
Module 15 of 15 in Practice and Strategy