Stage 1 summary. Foundations

Data is an observation recorded as a symbol: a number, a word, a pixel value, a sensor voltage, a timestamp. What makes something data is its status rather than its form, because it has been captured and not yet interpreted. The three structure classes sort the recording, not the meaning. Structured data sits in declared fields with declared types. Semi-structured data carries its shape with it, as in a JSON payload where structure travels with the values. Unstructured data such as a clinical letter or a call recording has no field layout at all.

A value becomes a fact only when the record says what it describes, in what unit, over what interval and under whose definition. A reading of 37.2 is nothing until something says degrees Celsius, this patient, this time. The same test defeats most headline claims about the volume of data in the world, since data created, captured and stored are different quantities and a forecast is not a measurement.

Russell Ackoff arranged data, information, knowledge, understanding and wisdom as a ladder in 1989, each level made by working on the one beneath. The four-level version this course uses is the later popularisation, which folds understanding into the step above it. A systolic reading of 145 is data. Naming the patient, the time and the baseline makes it information. Reading that against a clinical threshold makes it knowledge, because a rule has been applied. Deciding whether to change treatment, weighing history, medication and what the person wants, is judgement no rule supplies.

The ladder shows that the steps exist and are separately expensive. It is not a process model, and treating it as one is where it does damage: nothing in a pipeline turns information into knowledge by itself, and no accumulation of records produces wisdom. The claim worth catching is the one about a platform that turns data into insight, which names the problem and skips everything inside it. The productive direction is downwards, from the decision to the judgement it needs, back to the records that would support it.

A bit holds one of two states and a byte is eight of them, giving 256 values. Two prefix systems describe larger quantities and they disagree. SI prefixes are decimal, so a kilobyte is 1,000 bytes, which is what drive manufacturers sell. IEC prefixes, standardised in IEC 80000-13:2008, are binary, so a kibibyte is 1,024. Operating systems long computed in binary while printing the decimal label, which is why a drive sold as 1 TB shows as roughly 931 of what the file manager calls gigabytes. Nothing was lost; two systems were compared. A mebibyte exceeds a megabyte by 4.8 per cent and a tebibyte a terabyte by about 10 per cent.

Notation is for reading rather than computing. Hexadecimal packs four bits into a digit, which is why two hex digits describe one byte and why colours, MAC addresses and memory addresses are written that way. Endianness decides byte order in a multi-byte value: big-endian first is network byte order, little-endian first is what x86 processors use. The same four bytes read in the wrong order are a different number, and no error is raised.

Each format trades something specific. CSV is readable everywhere and carries no types, no nesting and no schema. JSON nests and types its values, and RFC 8259 requires UTF-8 for exchange outside a closed ecosystem, so encoding is a contract term. XML carries namespaces and an XSD schema, which is why it still runs financial and healthcare messaging. Parquet and Avro embed the schema in the file: Parquet stores column by column for analytical reads, Avro record by record for streaming.

Layout decides analytics performance. A row layout keeps one record's fields together and suits whole-record reads and writes. A columnar layout keeps one field's values together, so a query touching three of forty columns reads three and repeated values compress hard. Arrow is the in-memory form of the same idea. Two traps carry forward: text read under the wrong encoding garbles rather than errors, and binary floating point cannot hold most decimal fractions exactly, so money belongs in a fixed-point type.

Data about data is true and gives no test for what belongs in the record. The definition with teeth works from the reader: metadata is the recorded answer to the questions a competent stranger must resolve before using a dataset safely. What is this, how is it arranged, where did it come from, how current is it, and what may I do with it. The word stranger does the work, because whoever built the dataset already holds the answers and will not be there in two years.

Those questions sort into three families. Structural metadata says how the data is arranged: fields, types, units, precision, permitted values, keys, encoding. Descriptive metadata says what it is: title, description, coverage, the population sampled, known limitations. Administrative metadata says who is answerable and on what terms: owner, licence, access conditions, lawful basis, retention period and version. Most of it is cheap only on the day the data arrives, which is why a usable record is written then and not at the end.

A data standard specifies one of three things, and knowing which saves an argument. Some standardise a format, such as RFC 4180 for CSV or ISO 8601 for dates. Some standardise a vocabulary, meaning the values a field may take, such as country or diagnosis codes. Some standardise an exchange: how two parties transfer a payload, validate it, version it and reject it. De jure standards are ratified by a body such as ISO, W3C or the IETF; de facto standards win by adoption and are formalised later, which is what happened to JSON.

Interoperability has layers, and most expensive failures come from mistaking one for another. Syntactic means two systems can parse the same bytes. Semantic means they attach the same meaning to the fields, codes and units. Operational means the exchange survives a real process, with versioning, validation, error handling and named ownership. Two systems both using JSON have agreed on syntax and nothing else. GeoJSON under RFC 7946 orders coordinates longitude first while most mapping tools present latitude first, so two conforming systems can place a point in the ocean.

A dataset is never high quality in the abstract. ISO 8000-8:2015 ties quality to requirements and ISO/IEC 25012:2008 models the characteristics that can be measured, so the question is fitness for the use its consumers intend. The Government Data Quality Framework, drawing on DAMA UK, names the six dimensions used day to day: accuracy, completeness, uniqueness, consistency, timeliness and validity. Validity asks whether a value conforms to its rule and accuracy whether it matches the world, so a postcode can be valid and belong to somebody else.

Fitness for purpose makes the same file adequate for one decision and unacceptable for another. Five per cent address errors cost a wasted mailing in a campaign and misstate a regulatory return. ISO/IEC 5259 adds what data bound for machine learning needs, because a training set carries requirements about representativeness and labelling that a reporting extract does not. Only the process that creates the data can stop producing a defect, which is where quality is owned.

The lifecycle runs from planning, through collection and ingestion, preparation and storage, use and processing, sharing and publication, to archival or destruction. Each stage carries a control that works only if applied there. Planning sets the purpose, lawful basis, retention rule and quality rules. Collection validates at source. Storage applies access control and monitoring. Use is bounded by the declared purpose. Sharing carries the licence and access conditions. Disposal is verified destruction rather than a flag on a row.

Copies are the risk running through all of it. Deleting a record from production while the value survives in backups, extracts, downstream systems and audit logs is not disposal, and reporting it as one is how an erasure obligation quietly fails. GDPR Article 5(1)(e) keeps personal data identifiable no longer than the purpose needs and names no fixed period, so the decision is yours to justify. Lineage records which sources contributed, which transformations ran and which reports consume the output. Stage 5 returns to retention as a legal obligation.

The visual system reads position, length, colour and shape in parallel before conscious attention engages, which is why a bar chart lands faster than the table behind it. Chart selection is therefore analytical. Bars compare categories by length. Lines show change over time by slope. Scatter plots show the relationship between two continuous variables. Histograms show one variable's distribution. Choropleth maps show a value by geography. Tufte's data-ink principle removes anything carrying no information.

A chart can be truthful about its numbers and still lead a reader wrong. A truncated axis turns a one per cent difference into a visual doubling, and bars are the worst place for it because length is the encoding. A dual axis manufactures a relationship by choosing two scales. A three-dimensional effect encodes one number as two dimensions. Accessibility is part of honesty: never carry a distinction by hue alone, hold contrast to the WCAG 2.2 thresholds, and label series directly. An honest caption states what is measured, over what period, from what source, and what the chart does not show.

The UK Government's Data and AI Ethics Framework sets out three overarching principles for work with data, automated decision-making and AI, transparency, accountability and fairness, and five specific actions teams take to meet them. They exist because the two tests come apart. A collection can satisfy a lawful basis, pass legal review and still be the wrong thing to do, because the people in the data would object if they understood it or because the outcome falls hardest on those least able to challenge it.

Bias enters at identifiable points: what was chosen to be measured, who ended up in the sample, how the label was defined, which past decisions were treated as ground truth, where the threshold was set, and how the output is used. The passport photo checker that rejected darker-skinned applicants at higher rates learned that from a historical set which under-represented them. A harms-first review runs before collection and asks who is affected, who is worst affected, what happens to them when the system is wrong, and how they would challenge it.