Retention and defensible deletion

The legal starting point is the storage limitation principle. It does not name a number, which is the single most misunderstood thing about it.

A retention schedule turns that principle into dated, assignable instructions. It lists the classes of data the organisation holds and, for each class, the purpose it is kept for, the person accountable for it, how long it is kept, what event starts the clock, and what happens at the end. Without one, the default behaviour of every storage system is to keep everything forever, which quietly accumulates both breach exposure and cost that nobody has priced.

Four columns do most of the work, and three of them are usually missing from the policy documents I see. The class is a set of records with a shared purpose, not a table and not a system: complaint records, employee payroll records, CCTV footage. The clock start is an event, not a row creation timestamp, and this is where schedules most often break. A tax record clock runs from the end of the accounting period it relates to, not from the day the invoice was raised. An employment record clock runs from the end of employment. A complaint record clock runs from closure of the complaint. If the platform only knows when the row was written, it cannot implement any of these without a business event to hang the clock on.

The third column is disposition, and it has more than one option. Deletion is the obvious one. Anonymisation is legitimate where the analytical value outlives the identifiable purpose, and it satisfies Article 5(1)(e) directly because the data is no longer held in a form permitting identification. Archive is the third, and it is where schedules go to die: moving a class to cold storage without attaching a further clock converts a retention decision into an indefinite hold with a lower monthly bill.

The fourth column is the source of the period. Some clocks come from statute, some from sector guidance such as the NHS Records Management Code of Practice for health and care records, some from limitation periods that determine how long a claim could still be brought, and some from nothing more than an internal decision about business need. All four are legitimate, but they carry different weight when challenged, and a schedule that does not record which one applies cannot be defended by anyone other than its author.

A worked row makes the shape concrete. Take customer complaint records held by a utility. The class is complaint records, the purpose is handling the complaint and defending any subsequent challenge to how it was handled, the owner is the head of customer operations, the clock starts at closure of the complaint rather than at first contact, the period is set by how long a claim could still be brought plus the regulator's expectations for the sector, and the disposition is deletion of the identifiable record with an anonymised summary retained for service improvement. Every one of those cells is a decision somebody made and can be asked to justify. Compare that with a policy line reading complaints: seven years, which answers none of the questions an auditor will ask and gives a platform team nothing it can implement.

The gates figure above shows the resulting shape. A class is created and the clock is set once. At the active use gate the question is whether the purpose is still live. At the retention clock gate the question is whether the period has run. Only then does disposition choose a method. Retention is a schedule a class walks, not a delete button someone remembers to press. The gate ordering carries a point that is easy to miss: a purpose can expire before the clock does, and it can also outlast it. When the purpose ends first, the organisation is holding data it no longer needs and the schedule should be shortened rather than defended. When the clock ends first but a live purpose remains, the schedule was derived from the wrong instrument and the purpose has to be re-examined, because continuing to hold on the strength of an expired period is exactly what storage limitation prohibits.

A legal hold is an instruction, issued when proceedings have begun or may begin, requiring documents that could be relevant to be preserved and routine deletion to be stopped for them. In the Business and Property Courts the duty is set out in Practice Direction 57AD. Paragraph 4.1 makes clear that the documents to be preserved include ones that would otherwise be deleted under a retention policy or in the ordinary course of business, and paragraph 4.2 states the obligation to suspend the relevant deletion processes for the duration of the proceedings and to notify employees and former employees who may hold disclosable documents the party does not otherwise have.

Read that alongside storage limitation and the tension is obvious. One duty says delete when the purpose has expired; the other says preserve. They are reconciled by scope and by evidence. The hold applies to a defined set of records for a defined reason, that reason is a legal obligation, and both the imposition and the release are recorded. What is not defensible is the response I see most often, which is to pause the deletion job across the entire estate because nobody can express the narrower scope in the platform.

The mechanism matters. A hold suspends the clock; it does not restart it. The override lane in the gates figure leaves the chain after the retention clock and rejoins at disposition once the hold lifts, and the time already run still counts. A record with five years of a six-year period elapsed, held for two years, is due for disposition one year after release, not six. Implementing this means a hold has to be a flag on a set of records with its own lifecycle, not a global switch and not a rewritten retention date.

Legal hold also constrains the other rights. The right to erasure is not absolute: Article 17 lists the grounds on which it applies, and separate provisions disapply it where processing is necessary for compliance with a legal obligation or for the establishment, exercise or defence of legal claims. A record under a properly imposed hold is normally in exactly that position. The correct response to an erasure request touching held records is to explain the refusal and its basis, not to delete quietly and hope the two systems never meet.

Start with the legal text, because it sets the standard the engineering has to meet.

The erasure impact figure above lays out the layers the request has to reach: the subject request itself, the master record in the primary system, copies and caches in reporting, development and test environments, derivations and aggregates such as reports, models and indexes, and last the backups and tape. Each layer is a different engineering problem, the two in the middle are where programmes stall, and the figure marks the last one as the layer that follows a different rule from the rest.

Copies first. A row in a transactional database is normally also present in one or more read replicas, in a change stream, in the warehouse it was loaded into, in any downstream mart built from that warehouse, and in whatever extract a team took for a one-off analysis and never cleaned up. This is precisely what data lineage exists to answer, and running impact analysisdownstream from the source table is how a deletion programme discovers its own scope. A team without lineage is not able to state where a person's data is, which means it is not able to state that it has been erased.

Derived data is the second problem and the one most often missed entirely. Aggregations may or may not be personal data depending on whether an individual can still be singled out from them. Caches and search indexes hold copies with their own eviction rules. A feature store may hold engineered attributes computed from the deleted record. Free-text notes turned into a vector embeddingand loaded into an index are still a representation of that person's information, and the index needs its own deletion path rather than a rebuild scheduled for next quarter. Model weights trained on the data are harder again: there is no general operation that removes one training example from a trained model, so the honest position is to record the limitation and to control what is used for training in the first place.

Backups are the third problem and they are genuinely hard. A backup is a point-in-time image, often immutable by design because immutability is what makes it useful against ransomware. There is no supported way to reach into it and remove one person's rows, and attempting to do so would destroy the property that makes it a backup. The ICO's guidance on the right to erasure is direct about this. It says the key issue is to put the backup data beyond use even if it cannot be immediately overwritten, that the backup must not be used for any other purpose, and that the individual must be told clearly what will happen to their data, including that it may remain in the backup environment until it is replaced on an established schedule. The practical consequence for a platform team is that the length of the backup cycle becomes part of the answer given to the individual.

Then there are storage-level surprises. Modern analytical tables built on object storage and an open table format implement a row deletion as a new snapshot, and the previous snapshots remain readable for the configured time-travel window. Deleting a row and leaving time travel at ninety days means the row is still queryable for ninety days. Streaming platforms behave similarly: a delete published to an append-only log is a tombstone appended after the original, and the original stays in the log until compaction and the topic retention setting remove it. In both cases the deletion is real and the timing is governed by a configuration setting that is usually owned by a different team from the one answering the request.

One technique cuts through several of these at once. Where data is encrypted with a key held per tenant or per subject, destroying the key renders every copy protected by it unreadable, including copies inside images that cannot be edited. This is worth designing in early, but it has to be honest about its limits: it only covers data that was encrypted under that key from the start, and it does nothing about plaintext derived values that were computed and stored elsewhere.

Put together, this is why a deletion capability has to be built rather than announced. The work items are recognisable engineering: a business event on every class so the clock has something to start from, a copy inventory kept current by lineage rather than by a spreadsheet, a deletion path for every store that holds a copy including the ones with their own retention settings, a scoped hold mechanism, and a disposal record that outlives what it describes. None of that is glamorous and none of it appears on a product roadmap unless somebody puts it there. The predictable sequence is that it gets funded after the first regulatory correspondence, at which point it has to be built under a deadline set by someone else.

It is also worth naming what deletion buys, because a programme argued purely on compliance tends to lose to the next feature. Every record held past its purpose is a record that can be breached, that has to be searched during discovery, that has to be migrated at the next platform change, that has to be assessed when a subject access request arrives, and that costs storage and query time in the meantime. Deleting on schedule reduces all five. The organisations that manage retention well tend to be the ones that stopped treating it as a legal chore and started treating it as estate hygiene with a legal deadline attached.

Defensible deletion is deletion carried out under a documented, consistently applied process that an organisation can stand behind when challenged. The claim it supports is that the process was designed, approved and followed, not that no copy survives anywhere in the universe. That distinction is not a weakening. It is what makes the standard achievable and testable, and it is the same logic that runs through the rest of data protection law: what is required is an appropriate, documented, reviewed decision rather than a guarantee of a physical outcome.

Four things have to be true for a deletion to be defensible. The rule existed before the deletion, in an approved schedule, so the deletion was not a response to the dispute. The rule was applied consistently to the class, because selective deletion of inconvenient records within a class is the pattern that turns a routine disposal into an adverse inference. Holds were checked before the deletion ran, so the process itself proves that nothing under preservation was destroyed. And the outcome was recorded: what class, what records, on what date, under what rule, by what process, with what exceptions.

The evidence trail figure above is that last point drawn out as a chain of four steps. The schedule sets the period for the class. A trigger event fires the deletion. Execution runs the deletion across the copies and the backups. The record is the audit log that shows all of it happened, with timestamps and a subject reference where that is lawful. The figure's callout puts the weight on execution, because that is the step that produces evidence at all: a schedule with nothing behind it generates no log, and a policy with no execution log is paper.

What the record has to capture is one of three outcomes, because a purpose expiring does not always end in a deletion. Either the class was deleted and the disposal logged, or a lawful hold was recorded together with the reason the clock is suspended, or an exception was documented with a review date, which is the only legitimate form of keep it for now. An exception without a review date is not an exception; it is an indefinite retention that has been given a form to fill in.

Note the asymmetry that makes this work. Audit logging of disposal has to survive the disposal. If the log of what was deleted is itself inside the class being deleted, the organisation has destroyed its own evidence, and the position after a well-run deletion programme becomes indistinguishable from the position after a cover-up. The disposal record is metadata about the deletion, kept under its own schedule, and it should contain no more personal data than the identifiers needed to show that a specific obligation was met.

Consistency deserves a second look, because it is the criterion that decides most disputes. A schedule applied to everyone in a class is a routine disposal. The same schedule applied to everyone except the three accounts that later turned out to matter is something a court will characterise differently, whatever the intention was. This is why the deletion has to be driven by an automated rule over a class rather than by a person exercising judgement record by record, and why the approval that matters is the approval of the schedule rather than the approval of each deletion. A team that can show the job ran unattended against a class definition approved eighteen months earlier is in a far stronger position than one that can show a careful, well-reasoned decision taken last week.

Data subject rights then become answerable rather than aspirational. A request arrives, the identity is matched, the inventory names the locations, the process runs, the holds are respected and the refusals explained, and the record of all of it exists. That is the difference between an organisation that has a retention policy and one that has retention.