Data Advanced

This level assumes you are comfortable with Foundations and Intermediate. The focus now is scale, complexity, mathematical reasoning, and strategic impact. The notes are written as my perspective on how senior data professionals and architects think when systems get serious.

Maths in data systems describes patterns, uncertainty, and change. Abstraction turns messy reality into numbers we can reason about. At small scale numbers feel friendly. At scale, tiny errors compound and variation matters more than a single “best guess”.

My opinion: people do not fear maths because it is hard. They fear it because it is often introduced without kindness. If I introduce a symbol, I will tell you what it means, and I will show you a concrete example before I move on.

Mean (average)

Definitions:

• $x_i$: the $i$-th value in your dataset
• $n$: number of values
• $\bar{x}$: mean of the values

Variance (spread)

Definitions:

• $\sigma^2$: variance
• $x_i - \bar{x}$: deviation from the mean

Probability distributions describe how likely values are. A fair coin has a distribution of 0.5 heads and 0.5 tails. Real data distributions are rarely symmetrical. Knowing the shape stops us from trusting averages blindly.

Maths ladder (optional). From intuition to rigour, properly explained

Foundations. Ratios, units, and “does this number even make sense”

Expert data work starts with units. A number without a unit is a rumour. If one system logs energy in kWh and another in MWh, the data can be perfectly stored and perfectly wrong. Simple check: write the unit beside the value and ask if the magnitude is plausible.

Next step. Dot product (a simple way to combine measurements)

Intuition: it combines two lists of measurements into one number. In modelling, dot products appear everywhere (for example, linear models).

Deeper. Bayes’ theorem (updating beliefs with evidence)

Bayes’ theorem connects conditional probabilities:

• $A$: a hypothesis (for example, “the alert is a real incident”)
• $B$: observed evidence (for example, “we saw a suspicious login pattern”)
• $P(A)$: prior belief about $A$ before seeing $B$
• $P(B \mid A)$: likelihood of seeing $B$ if $A$ is true
• $P(B)$: overall probability of seeing $B$

Why it matters: many “data decisions” are really belief updates under uncertainty. The maths keeps you honest about what evidence can and cannot justify.

Rigour direction (taste). Error propagation and compounding uncertainty

In real pipelines, uncertainty compounds. A value is measured, transformed, aggregated, and modelled. If each step adds error, you can end up with a final metric that looks precise but is not.

The serious lesson: do not only track “numbers”. Track how the numbers were produced and what could distort them. This is why lineage and verification exist.

Models are simplified representations of reality. They exist so teams can agree on how data fits together. Abstraction hides detail to make systems manageable. The risk is that hidden detail was needed for a decision you care about.

Entity relationships show how things connect. Customers place orders, orders contain items. Dimensional models separate facts (events) from dimensions (who, what, when). Simpler models are easy to query but may miss nuance. Richer models can be harder to govern.

Design trade offs are unavoidable. A lean model may skip location because it is not needed today. Later, when someone asks about regional patterns, the model cannot answer. Bias also hides in models: if a field is dropped, whole groups can disappear from analysis.

Worked example. The field you delete is the question you cannot answer later

A team drops location data “because it is messy”. Six months later, an incident requires regional analysis. The team scrambles for ad-hoc extracts and guesses, because the model made the question impossible.

My opinion: data models are long-term commitments. When you drop a field, you are not only simplifying. You are deciding which questions future you is not allowed to ask.

Verification. Check your model before you build on it

• Name three questions your model must answer today.
• Name one question it should answer in six months, even if you do not need it yet.
• Identify one field that is high risk (personal data, sensitive attributes) and state how it will be protected or removed.

Inference is about drawing conclusions while admitting uncertainty. Correlation means two things move together. Causation means one affects the other. Mistaking correlation for causation leads to confident but wrong decisions.

Sampling takes a subset of the population. If the sample is biased or too small, the answer will drift from reality. Confidence is how sure we are that the sample reflects the population. Errors creep in when data is noisy, samples are skewed, or models are overconfident.

Statistics is humility with numbers. Every estimate should come with a range and a note on what could be wrong.

Common mistakes (the expensive ones)

• Treating “statistically significant” as “important”, without checking effect size and practical impact.
• Running many comparisons until something looks exciting, then stopping.
• Treating a model score as truth rather than a measurement with uncertainty and failure modes.
• Reporting a single number without showing distribution or tail behaviour.

Data systems distribute to handle scale and resilience. Latency is the time it takes to respond. Consistency is how aligned copies of data are. Failures are normal in distributed systems, so we plan for them instead of hoping they do not happen.

A simple intuition: you can respond fast by reading nearby copies, but those copies might be slightly out of date. Or you can wait for all copies to agree and respond slower. Users and use cases decide which trade off is acceptable.

Worked example. The dashboard says “yesterday”, but the decision is “now”

Eventual consistency can be perfectly acceptable for a monthly report. It can be unacceptable for fraud detection, outage response, or operational dispatch.

My opinion: the right question is not “is it consistent”. The right question is “consistent enough for which decision, at which time”.

Regulation exists to protect people and markets. Accountability means someone can explain what data is used, why, and with what safeguards. Auditability means we can trace who did what and when. These are not just legal boxes. They build trust with users and stakeholders.

Ethics and trust sit beside regulation. If a decision harms people, compliance alone is not enough. Long term consequences include fines, loss of reputation, and slower delivery because teams stop trusting data.

Governance at scale. A practical view of DAMA style coverage

Many organisations use a DAMA DMBOK style lens to describe data management capabilities. I treat it as an orientation map, not scripture. The useful part is that it forces you to look at the whole system, not only the warehouse.

Common mistakes (enterprise governance edition)

• Calling something “governed” because there is a document, not because controls are enforced.
• Creating committees with no decision rights, then wondering why teams route around them.
• Treating metadata as optional. When incidents happen, metadata becomes the evidence trail.

Verification. A defensible explanation a regulator would accept

• Write one paragraph explaining what the dataset is for, who can access it, and why.
• State what would trigger an investigation (unexpected access, unusual exports, anomalous changes).
• Describe one control that reduces harm, not only one that reduces paperwork.

Data creates value when it improves decisions, products, and relationships. Network effects appear when sharing makes each participant better off. Competitive advantage comes from combining quality data with disciplined execution, not from hoarding alone.

Monetisation can be direct (selling insights) or indirect (better products). Lock in can help or hurt: it keeps customers, but it can also trap you with legacy systems. Long term risk comes from overcollecting, underprotecting, or failing to renew data pipelines.

Data as a product (the difference between reuse and “please send me the extract”)

If every request becomes a one-off extract, you are not running a data capability. You are running a bespoke reporting service. A data product is a dataset with an interface, documentation, quality guarantees, and an owner. It is designed for consumers.

Data mesh, used properly (not as a slogan)

Data mesh is a response to a real organisational problem: central teams become bottlenecks because domains do not own their data. The useful idea is domain ownership plus platform support plus federated governance. The dangerous version is “every team does whatever they want”.

Worked example. A data mesh that failed because the platform was missing

Leaders announce “data mesh”. Domains are told to publish data products. There is no shared platform, no templates, and no quality tooling. Domains publish inconsistent datasets and consumers lose trust.

My opinion: you cannot decentralise responsibility without centralising enablement. If you want domain ownership, you must provide a self-serve platform and a small set of enforced standards.

Verification. Strategy that is not just motivational posters

• Name one value outcome (time saved, risk reduced, revenue protected) and how you would measure it.
• Name one dependency (people, platform, governance) that could block the value.
• Write one uncomfortable trade-off you will accept, and why.

At this level, your evidence should show judgement under uncertainty. You are no longer proving you can repeat definitions. You are proving you can make trade-offs, explain them, and defend them.

• What I studied: advanced maths foundations, inference, distributed trade-offs, governance, and strategy.
• What I applied: one concrete decision. Example: “I chose eventual consistency for reporting but required stronger guarantees for operational alerts.”
• What could go wrong: one failure mode and the control that would detect it early.
• Evidence artefact: a short decision record (ADR-style) with assumptions, trade-offs, and verification steps.