Module 4 of 52

Data representation and formats

How computers encode characters, numbers, images, and structured records, and why format and encoding choices have real consequences for data quality.

By the end of this module you will be able to:

  • Explain why UTF-8 dominates web encoding and how it differs from ASCII
  • Describe the IEEE 754 floating-point precision problem with a concrete financial example
  • Compare JSON, XML, and CSV for a given use case
  • Distinguish lossless from lossy compression and identify appropriate uses for each

Format boundaries: where CSV, JSON, Parquet, and RDF hit their limits

Read across the schema row and the four formats split: CSV leaves it implicit in the header and JSON makes it optional, while Parquet embeds it and Turtle carries vocabularies, so the two loose formats hand schema discovery to whoever reads the file next.

Format choice is a five-property decision: native shape, schema, validation, compression and typing, and fit for purpose. RFC 4180 CSV is row-shaped; RFC 8259 JSON is tree-shaped; Parquet adds columnar typing; W3C RDF Turtle is graph-shaped.

Format choice is a five-property comparison across four shapes Matrix. Header row names the property column plus CSV, JSON, Parquet, RDF Turtle columns. Five property rows: Native shape, Schema (emphasised in red soft), Validation, Compression and typing, Best fit. Each cell answers the property for that format. A red-accent callout names schema as the boundary that matters most: CSV and JSON make it optional; Parquet and Turtle bind it in. FORMAT BOUNDARIES · WHERE EACH SHAPE HITS ITS LIMIT PROPERTYCSVJSONParquetRDF Turtle Native shapeFlat rowsTreesTyped columnsGraph of triples SchemaImplicit, by headerOptional, JSON SchemaEmbedded + strictVocabularies + SHACL ValidationBy conventionTooling levelBuilt-inSHACL shapes Compression / typingNoneText + UTF-8Columnar + zstdText + UTF-8 Best fitSpreadsheet exchangeWeb APIsAnalytics warehousesLinked open data The boundary that matters most: schema CSV and JSON make schema optional, so consumers re-discover it after the fact. Parquet and Turtle bindschema in, which is why they win for analytics and linked data.

The same three records laid out for rows and laid out for columns

The same two-field query touches eighteen cells under row layout and six under columnar, because rows keep a record together and columns keep a field together, which is why wide scans go columnar and single-record writes stay in rows.

Row layout keeps a record together, so a two-field query still reads all six fields of every record. Columnar layout keeps a field together, so the same query reads only what it asked for, which is why scans go columnar and single-record writes stay in rows.

The same three records laid out for rows and laid out for columns A query card at the top reads two of six fields, and two elbow arrows drop from it into two lanes. The left lane, row layout, draws the three records as three horizontal strips, each strip holding all six fields together: id, time, kwh, site, rate, flag. Every cell is tinted because all of them are read, and the two wanted fields carry a deep red border. It reads all 18 cells to return 6. The right lane, columnar layout, draws the same records as six vertical field groups, each holding r1, r2 and r3 together; only the time and kwh groups are tinted. It reads 6 cells to return 6. A legend explains the tints and two chips name the trade. SAME THREE RECORDS · TWO STORAGE LAYOUTS THE QUERY Read two of six fields reads all six fields of every record reads two fields only ROW LAYOUT one record stays together Reads all 18 cells to return 6 r1 id time kwh site rate flag r2 id time kwh site rate flag r3 id time kwh site rate flag COLUMNAR LAYOUT one field stays together Reads 6 cells to return 6 id r1 r2 r3 time r1 r2 r3 kwh r1 r2 r3 site r1 r2 r3 rate r1 r2 r3 flag r1 r2 r3 read from storage the two fields the query wants not read at all Row layout wins the single-record write Columnar wins the wide scan

How character encoding either preserves or corrupts text

Both rows send identical characters, code points and bytes, and they part only at the receiver, so the corruption in the lower row comes from an unstated encoding rather than from anything that happened to the data in transit.

Character encoding is the contract that lets the receiver read what the sender wrote. A mismatch turns valid bytes into mojibake. Unicode Standard 15.1 §3 separates the character from the code point and the bytes; RFC 8259 §8 makes UTF-8 the JSON default.

Encoding agreements decide whether characters survive exchange Two parallel four-stage flows. Top row SUCCESS shows Character -> Code point -> Bytes -> Receiver reading the same character; the final Receiver card is emphasised in red soft. Bottom row FAILURE shows the same flow but the final card reads Receiver expects ISO-8859-1 and the body shows mojibake. Brand-red arrows step through both flows. A red-accent callout names UTF-8 as the only safe default. CHARACTER ENCODING · CONTRACT BETWEEN SENDER AND RECEIVER SUCCESS · ENCODING MATCHES 1 Character A or é or 中 2 Code point U+0041, U+00E9,U+4E2D 3 Bytes UTF-8 encoded stream 4 Receiver Same character read FAILURE · ENCODING MISMATCH 1 Character A or é or 中 2 Code point U+0041, U+00E9,U+4E2D 3 Bytes UTF-8 encoded stream 4 Receiver expectsISO-8859-1 A or é or äø- State the encoding next to the file UTF-8 is the only safe default. CSVW §5 and RFC 8259 §8 both make this explicit; an unstated encodingis a data-loss waiting to happen.

Character encoding: ASCII, UTF-8, and Unicode

A character encoding is a mapping between characters (letters, digits, symbols) and their binary representations. ASCII (American Standard Code for Information Interchange, standardised 1963) encodes 128 characters using 7 bits: the 26 uppercase and 26 lowercase Latin letters, digits 0-9, punctuation, and 33 control characters. It cannot represent accented characters, non-Latin scripts, or even the pound sign (£).

UTF-8 (Unicode Transformation Format, 8-bit) is a variable-width encoding of the Unicode character set. It is backward-compatible with ASCII for the first 128 characters but uses 2 to 4 bytes for characters outside that range. The pound sign £ (Unicode U+00A3) encodes as two bytes in UTF-8 (0xC2 0xA3). In the legacy Windows-1252 encoding, it encodes as a single byte (0xA3). A file saved as Windows-1252 and read as UTF-8 produces garbled output because 0xA3 in UTF-8 is a continuation byte in a multi-byte sequence, not a standalone character.

The web has largely converged on UTF-8 because it can represent the full Unicode character set while preserving ASCII compatibility for the first 128 code points. JSON exchanged outside a closed ecosystem must be encoded as UTF-8 under RFC 8259, so encoding is a contract requirement, not a display preference.

Text errors come from treating bytes as characters without the right encoding. Numeric errors come from treating approximations as exact values.

Numbers: IEEE 754 floating-point and decimal precision

Floating-point numbers use the IEEE 754-2019 standard. A 32-bit float uses 1 sign bit, 8 exponent bits, and 23 mantissa bits. Floating-point numbers cannot represent most decimal fractions exactly in binary: 0.1 is a repeating binary fraction. This means 0.1 + 0.2 in floating-point arithmetic does not equal exactly 0.3. For financial calculations, always use fixed-point decimal types (SQL DECIMAL, Python's decimal.Decimal), not floating-point.

Choosing a numeric type settles what a single value means. The next choice is the container that carries whole records between systems.

Structured data formats: CSV, JSON, XML, and binary alternatives

Three formats dominate structured data exchange:

  • CSV: rows of comma-separated values, human-readable, universally supported. Weaknesses: no type information, no nesting, ambiguous handling of commas within values.
  • JSON: supports nesting, arrays, and typed values (strings, numbers, booleans, null). The dominant REST API format. Required by RFC 8259 to be UTF-8 encoded.
  • XML: supports schemas (XSD), namespaces, and complex document structures. More verbose than JSON. Prevalent in legacy enterprise systems, financial messaging (SWIFT, ISO 20022), and healthcare (HL7 v2/v3).

Apache Parquet and Apache Avro are binary formats optimised for large-scale analytics. Not human-readable, but offer far better compression and query performance than text formats. Parquet is columnar, enabling efficient queries that access only specific columns in large datasets.

Once values are represented, systems still need to move and store them efficiently. Compression is safe only when the loss model matches the data.

Compression: lossless and lossy

Lossless compression allows perfect reconstruction of the original data from the compressed version. No information is discarded. Examples: DEFLATE (used in ZIP and PNG), LZ4, Zstandard. Appropriate for text, code, databases, and any data where integrity is essential.

Lossy compression permanently discards some information to achieve higher compression ratios. The original data cannot be fully reconstructed. Examples: JPEG (images), MP3 (audio), H.264 (video). Appropriate for media where small quality reductions are imperceptible to human senses.

A 24-megapixel RAW photograph might be 24 MB. As a high-quality JPEG it might be 4 MB (6:1 ratio) with no visible quality loss. As a low-quality JPEG it might be 400 KB (60:1 ratio) with visible artefacts. PNG lossless would be approximately 18 MB, preserving every pixel.

Never apply lossy compression to structured data. Applying JPEG compression to a CSV file would corrupt the data silently. Re-saving a JPEG introduces additional quality loss each time; this is why document archives should use lossless formats.

Common misconception

“JSON is just text, so there are no encoding issues as long as the JSON syntax is valid.”

RFC 8259 requires JSON to be encoded in UTF-8. Many real-world JSON producers emit Windows-1252 or ISO 8859-1 encoded files with a .json extension. Parsers that do not validate encoding before parsing will either fail on non-ASCII characters or produce incorrect character data silently. If a supplier sends JSON containing French accented characters (such as in city names) and it was saved in Windows-1252, your UTF-8 parser will raise an error or corrupt those characters. Always validate encoding before parsing, and require UTF-8 explicitly in API contracts.

Check your understanding

A financial system stores transaction amounts as 32-bit single-precision floats. A transaction for £99.99 is stored and then retrieved. The system displays £99.98999786376953. What is the root cause?

A data engineer receives a JSON file from a third-party supplier containing French city names with accented characters such as Nimes. The engineer's parser raises an error on several records. The file has a .json extension. What is the most likely cause?

Core distinctions

  • ASCII encodes 128 characters in 7 bits; UTF-8 extends this to the full Unicode character set using variable-width encoding and is backward-compatible with ASCII.
  • IEEE 754 floating-point cannot represent most decimal fractions exactly. Financial data must use fixed-point decimal types (SQL DECIMAL, Python's decimal.Decimal). The 0.1 + 0.2 != 0.3 problem is a direct consequence.
  • JSON, CSV, and XML have different trade-offs: CSV is compact and universal but lacks types and nesting; JSON supports nesting and is the dominant API format; XML is verbose but schema-capable, used in finance and healthcare.
  • Lossless compression (DEFLATE, LZ4) preserves data exactly. Lossy compression (JPEG, MP3) discards information permanently. Never apply lossy compression to structured data files.

Knowing how data is encoded and formatted prepares you for the next question: how do systems agree on what formats to use? The next module covers data standards and interoperability, from ISO 8601 dates to HL7 FHIR health records, and the real-world cost of getting it wrong.

Standards and sources cited in this module

  1. RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format

    Section 8.1: UTF-8 encoding requirement for JSON. The formal specification of JSON syntax.

  2. IEEE 754-2019: Standard for Floating-Point Arithmetic

    Floating-point representation format: single precision (32-bit) and double precision (64-bit) specifications.

  3. Unicode Standard 17.0 (2025)

    Current Unicode standard version. Chapter 2 covers the general structure of Unicode encoding.

  4. WHATWG Encoding Standard

    Living standard for web encoding behaviour, including UTF-8 handling and legacy label compatibility.

Back: Units, notation and binaryNext: Metadata and context