Digitalisation Intermediate

This level turns foundations into applied design. We move from ideas into data flows, integration choices, and the signals you must monitor in real operations.

Data pipelines and flows

I always sketch flows first: where the data starts, where it stops, and how it becomes useful. That makes governance practical, not theoretical.

🧪

Worked example. The pipeline that “worked” until month-end

A pipeline runs daily, dashboards look fine, everyone relaxes. Then month-end hits: volumes spike, late data arrives, and the transformation step times out. People then argue about “data quality” when the real issue is that nobody designed the pipeline for peak load and backfill.

⚠️

Common mistakes in pipeline design

• Assuming “daily” means “stable”. Seasonality and operational events will break you.
• No ownership for each stage (ingest, validate, store, transform, serve).
• No backfill strategy. Late data becomes silent corruption.
• No “stop the line” behaviour when validations fail.

🔎

Verification. Can you operate it at 2am

• If a validation fails, what happens. Does the pipeline stop, quarantine, or silently continue.
• If a source system replays events, do you get duplicates.
• If a field changes, do you alert downstream consumers.
• Can you answer: what changed, when, and who approved it.

📝

Reflection prompt

Think of one data flow you rely on. Where would it hurt most if it was wrong for one week without anyone noticing.

Analytics, AI, and control loops (where digitalisation becomes “active”)

Collecting data is the easy part. The hard part is turning it into action safely. Analytics and AI help you detect patterns and predict outcomes. Control loops are how the system responds.

🧪

Worked example. A smart rule that caused oscillation

A team builds an automated rule: “if demand is high, reduce load by sending a message to flexible devices”. Many devices respond at the same time. Demand drops sharply. The rule then stops. Devices recover. Demand spikes again. The system starts oscillating.

My opinion: automation without systems thinking is how you create elegant chaos. You need rate limits, damping, and measurement to prevent the control loop from fighting itself.

⚠️

Common mistakes with analytics and automation

• Treating model output as a decision, not an input to a decision.
• No monitoring for drift, so performance degrades quietly over time.
• No safety boundaries. The system can do the wrong thing quickly and at scale.

🔎

Verification. A safe automation checklist

• What is the objective, and how do you measure success.
• What is the failure mode, and how do you detect it early.
• What is the human override, and who can trigger it.
• What is the rollback, and have you rehearsed it.

📝

Reflection prompt

Think of one automation you would not allow to run without a human approval step. Why that one.

APIs and system integration

When integrations are vague, every team invents their own definition and version. The result is slow change and fragile services.

🧪

Worked example. A breaking change that nobody meant to ship

A team adds a new required field to an API request because “we need it now”. In their service, it works. Downstream, consumers fail, retries spike, and suddenly your “integration” is a denial of service against yourself.

⚠️

Common mistakes in integration

• Treating APIs like internal functions instead of contracts with external consumers.
• No versioning strategy, so “small changes” become outages.
• Undefined error semantics. Consumers then guess and build brittle retry loops.
• No idempotency, so retries create duplicates and weird state.

🔎

Verification. A minimal contract quality bar

• Explicit versioning policy (and a deprecation window).
• Stable identifiers and idempotency where retries can happen.
• Error codes and error bodies that consumers can act on.
• Example payloads that match reality, not optimism.

📝

Reflection prompt

If you had to support this API for five years, what is the one change you would be scared to make. That fear is telling you what you should design properly now.

Data models and mapping

Mapping is a design choice. It decides what is essential, what can be dropped, and what must stay consistent across services.

🧪

Worked example. The mapping that “looked right” and quietly rewrote history

A source system stores “status” as free text. The canonical model uses a strict enum. A mapper converts unknown values to “other” to keep the pipeline running. It feels helpful until you realise you have destroyed the ability to answer “what really happened” later.

⚠️

Common mistakes in mapping

• Mapping based on field names instead of meaning.
• Dropping “unknown” values instead of quarantining and fixing upstream.
• No mapping tests, so changes slip in with good intentions and bad outcomes.
• No steward for the canonical model, so definitions drift.

🔎

Verification. Mapping checks that prevent nonsense

• Coverage: what percentage of records map cleanly.
• Exceptions: unknown values listed and reviewed, not hidden.
• Round-trip: can you map forward and still explain the original meaning.
• Versioning: mapping rules have versions and change logs.

📝

Reflection prompt

Name one field that should never be silently “best guessed”. What should the system do instead when it cannot map it safely.

Operations, monitoring and observability

Monitoring must cover both speed and safety. If you only watch speed, you miss quality. If you only watch quality, you miss delivery friction.

🧪

Worked example. The dashboard that looked fine while users suffered

A dashboard shows average response time is stable. Meanwhile, a small percentage of users hit a slow path and abandon the journey. The team says “the service is healthy” because the average is comforting. Users do not experience averages.

⚠️

Common mistakes in observability

• Averages only. Percentiles tell you about the bad day.
• No link between signals and action. Alerts fire, nobody owns response.
• Logging without correlation IDs, making root cause analysis slow.
• No business signals. You watch latency but miss drop-off and rework.

🔎

Verification. Your minimum operational pack

• Request rate, error rate, and latency percentiles for key endpoints.
• Journey drop-off and “contact us” rate by step.
• Data pipeline freshness and validation failure rate.
• A clear on-call path and a rollback plan that you have rehearsed.

🧾

CPD evidence (practical, not performative)

• What I studied: pipelines, contracts, mapping, and operational signals.
• What I practised: one mapped data flow with owners, one contract review, and one monitoring pack for a journey.
• What changed in my practice: one habit. Example: “I will ask for the error semantics before I sign off an integration.”
• Evidence artefact: a one-page diagram of a pipeline plus a monitoring checklist.