Data lifecycle and flow
By the end of this module you will be able to:
- Identify the GOV.UK data lifecycle stages for a described dataset
- Apply the GDPR storage limitation principle to a retention decision
- Explain why data lineage matters for audit and troubleshooting
- Describe what metadata should be captured at each lifecycle stage
Six lifecycle control points wired as a closed loop
Every stage carries a control that travels with the data, from a purpose stated at Plan to erasure evidenced at Retire, and the sixth arrow closes the loop because that retirement evidence is what the next Plan is built on.
The data lifecycle is a closed loop because retirement evidence feeds the next plan. Every transition carries a lawful-basis or lineage control. UK GDPR Article 5 makes you able to demonstrate that control, and W3C PROV-DM gives the who, what, when and why a shared shape, so the record can be produced on demand.
Why every shadow copy is a new retention liability
A sanctioned copy carries a named purpose and an expiry date; a shadow copy carries neither, so nothing ever fires to remove it and it sits on a local drive for years as an unbounded retention risk.
Every unauthorised copy of personal data is a fresh retention risk. UK GDPR Article 5(1)(e) requires data kept no longer than necessary; ISO/IEC 27701:2025 §5.5 requires the controller to track and bound every copy.
Lifecycle risk review as an ordered audit path
The review runs register, classify, schedule and retire in that order, and the schedule step is what makes the other three real: a risk tier with no review date in the calendar ages quietly, which is why the missed review is what an audit finds.
A lifecycle risk review walks four steps: register every dataset, classify by risk, schedule the next review, retire with evidence. UK ICO data sharing code requires this loop for personal data; DAMA-DMBOK 2 generalises it to all governed datasets.
Every piece of data has a lifespan. GOV.UK frames that lifespan as planning, collection or ingestion, preparation and maintenance, use, sharing, and eventual archival or destruction. Managing these stages is not bureaucracy. It is how organisations prevent quality defects, meet regulatory obligations, control costs, and maintain trust with the people whose data they hold.
Lifecycle discipline starts before data is collected. Planning determines what data is needed, what standards will be used, which metadata must be captured, and how deletion will happen later.
9.1 The six stages
The data lifecycle is the sequence of stages through which data passes from initial planning to its eventual archival or destruction:
- Plan: decide the business need, source, standards, lawful basis, retention period, and quality rules before collection starts.
- Collect or acquire, and ingest: data is generated, received, validated at source, and brought into controlled systems.
- Prepare, store and maintain: data is formatted, linked, cleaned, protected, monitored, and kept fit for use.
- Use and process: data is queried, transformed, analysed, or fed into models for a defined purpose.
- Share and publish: data is distributed to authorised consumers or published with quality assurance, documentation, and the right access conditions.
- Archive or destroy: data is retained securely with its quality context, or securely deleted when there is no continuing purpose or legal basis.
Each stage carries distinct metadata requirements, quality considerations, and governance obligations.
Click through each stage in the interactive diagram below to see what happens, what metadata to capture, and what risks arise when a stage is neglected.
Retention cannot be fixed at the end of the lifecycle if it was never planned. The deletion rule has to be known before downstream copies, backups, extracts, and archives multiply.
9.2 Retention and deletion obligations
The retention period is the defined length of time that data must be kept before it can or must be deleted. Periods may be set by law, regulation, contract, or internal policy. UK organisations navigate multiple overlapping frameworks.
“kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed;”
UK GDPR, Article 5(1)(e) - Article 5(1)(e), Storage limitation
There is no single prescribed retention period in GDPR. The period depends on purpose. HMRC recommends six years for most financial records with personal data. NHS health records have longer prescribed periods under the NHS Records Management Code. The key principle: keep data only as long as the purpose requires, then delete it.
ISO/IEC 27001:2022 Annex A.8.10, Information deletion, requires information held in information systems, devices or any other storage media to be deleted once it is no longer required. A separate control, Annex A.7.14, covers equipment being disposed of or re-used: storage media must be verified so that sensitive data and licensed software have been removed or securely overwritten first. Deleting securely means more than pressing delete: it means verifiable, documented destruction of all copies.
Common misconception
“Deleting a row from the production database means the data is gone.”
Deleting from production while retaining copies in backups, audit logs, disaster recovery sites, and data warehouse extracts does not constitute compliant disposal under GDPR. A deletion programme must inventory all locations where the data exists (production, backups, archives, downstream systems) and address every instance before marking records as destroyed.
Deletion decisions require a map of where the data has moved. Lineage provides that map for audit, troubleshooting, impact analysis, and subject rights work.
9.3 Data lineage
Data lineage is a record of a dataset's origins, transformations, and movements over time. It documents which source systems contributed, which transformations were applied at each stage, and which downstream systems or reports consume the output. Lineage enables three practical capabilities:
- Debugging: when a report shows an unexpected value, lineage allows tracing the value back through transformations to the source.
- Regulatory compliance: under GDPR Article 14, data subjects have the right to know the source of data held about them. Under FCA regulations, firms must demonstrate the provenance of data used in regulatory reporting.
- Impact analysis: before changing a source system's schema, lineage shows which downstream datasets and reports will be affected.
Modern data catalogue tools (Apache Atlas, Collibra, Alation) capture lineage automatically by monitoring SQL queries and ETL (Extract, Transform, Load) operations.
“from which source the personal data originate, and if applicable, whether it came from publicly accessible sources;”
UK GDPR, Article 14(2)(f) - Article 14(2)(f), Information to be provided where personal data have not been obtained from the data subject
This right means organisations must know where their data came from. Without lineage records, answering a data subject access request (DSAR) becomes guesswork. Lineage is not just a technical nice-to-have; it is a regulatory requirement for personal data.
Lineage is one form of metadata. The wider metadata record explains purpose, source, quality, access, retention, and the limits of safe reuse.
9.4 Metadata at every stage
Metadata is data about data. It is essential at every lifecycle stage for discoverability, quality assessment, and compliance. The interactive diagram above lists specific metadata requirements per stage. The overriding principle: capture metadata at the point of creation, not retrospectively.
Common misconception
“We can add metadata later when we build the data catalogue.”
Metadata that is not captured at creation time is extremely difficult to reconstruct. Legacy datasets without recorded origin, purpose, or consent basis cannot be compliantly used, shared, or deleted. Retrospective metadata reconstruction projects often cost more than the data is worth. The time to capture metadata is when the data first enters the organisation.
A GP surgery collects patient consultation notes containing personal and special category health data. A practice manager is unsure whether to retain the notes for 5 years, 10 years, or indefinitely. Which GDPR principle is most directly relevant?
An analyst discovers that a revenue figure in the monthly board report is incorrect. The report was generated from a data warehouse loaded by a pipeline sourcing three CRM systems. What is the most efficient way to find where the error was introduced?
Your organisation deletes customer records from the production database when they close their account. An internal audit discovers the same records still exist in nightly backups, the data warehouse, and two downstream reporting extracts. Is the deletion GDPR-compliant?
Core distinctions
- The GOV.UK data lifecycle runs through plan, collect or acquire and ingest, prepare store and maintain, use and process, share and publish, and archive or destroy. Each stage has distinct metadata, quality, and governance requirements.
- GDPR Article 5(1)(e) (storage limitation) requires personal data to be deleted when it is no longer necessary for its collection purpose. Retention periods must be documented and enforced through automation.
- Data lineage records the origin, transformations, and movements of a dataset. It enables debugging, regulatory compliance (GDPR Article 14), and impact analysis before schema changes.
- Metadata captured at creation time is essential and nearly impossible to reconstruct later. The time to record source, purpose, and consent basis is at the point of collection.
- Deletion means addressing all copies: production, backups, archives, data warehouses, and derived datasets. A row deleted from production is not GDPR-compliant if copies persist elsewhere.
Standards and sources cited in this module
Article 5(1)(e) (Storage limitation), Article 14(2)(f) (Source disclosure)
Storage limitation principle governing retention and the right to know data sources. Both drive lifecycle management practices.
Annex A.8.10 (Information deletion), Annex A.7.14 (Secure disposal or re-use of equipment)
A.8.10 requires information to be deleted once it is no longer required. A.7.14 requires storage media to be cleared before equipment is disposed of or re-used.
NHS Records Management Code of Practice (2021)
Section 4 (Retention schedules)
Current NHS England Digital landing page for health and care retention guidance, including updates to Appendix II.
Storage limitation chapter
ICO interpretation of Article 5(1)(e) with practical examples for UK organisations.
Open standard for lineage metadata collection
Modern lineage model for datasets, jobs, and runs. Useful for operational debugging, impact analysis, and audit evidence.
Module 8 of 52 · Foundations