Data Formats & Wire Protocols

JavaScript Object Notation (JSON) revolutionized modern web computing, supplanting cumbersome XML as the dominant format for REST APIs, configuration files, and NoSQL databases. Specified by Douglas Crockford in 2001, JSON defines six core data types: objects (key-value maps), arrays (ordered lists), strings, numbers (IEEE 754 float), booleans, and null. Its simplicity, human readability, and ubiquitous native language runtimes make it the definitive data serialization standard of the Internet.

Click Blueprint to Enlarge

{
  "protocol": "HTTPS",
  "version": 3,
  "endpoints": [
    {
      "path": "/api/v1/metrics",
      "rateLimit": 1000,
      "authRequired": true,
      "payload": {
        "format": "json",
        "compression": ["zstd", "br"]
      }
    }
  ],
  "active": true
}

JSON Lines (JSONL), or Newline-Delimited JSON (NDJSON), addresses the greatest limitation of standard JSON for big data: parsing massive arrays requires loading the entire file into memory before closing brackets can be matched. JSONL structures files such that each single line is a valid, independent JSON object separated by a newline (\n). This unlocks infinite streaming, parallel chunking, and append-only operations without file corruption.

Click Blueprint to Enlarge

{"id": "conv-001", "role": "system", "content": "You are a database optimizer."}
{"id": "conv-002", "role": "user", "content": "Explain LSM compaction strategies."}
{"id": "conv-003", "role": "assistant", "content": "Size-tiered vs leveled compaction optimize write vs read amplification."}

Extensible Markup Language (XML) is a W3C standard designed for transmitting structured data and document formatting with rigorous semantic metadata. With support for tag attributes, namespaces (avoiding name collisions), comments, CDATA blocks, and robust schema validation via XSD (XML Schema Definition), XML powered enterprise computing, SOAP web services, SVG graphics, Office documents (DOCX, XLSX), and publishing pipelines for decades.

Click Blueprint to Enlarge

<?xml version="1.0" encoding="UTF-8"?>
<Document xmlns="urn:iso:std:iso:20022:tech:xsd:pain.001.001.03">
  <CstmrCdtTrfInitn>
    <GrpHdr>
      <MsgId>TX-2026-9042</MsgId>
      <CreDtTm>2026-10-10T12:00:00Z</CreDtTm>
      <NbOfTxs>1</NbOfTxs>
    </GrpHdr>
    <PmtInf>
      <Amt Ccy="USD">4500.00</Amt>
      <Cdtr><Nm>High-Performance Corp</Nm></Cdtr>
    </PmtInf>
  </CstmrCdtTrfInitn>
</Document>

YAML is a human-friendly data serialization language meticulously designed for readability and maintainability. Using Python-style whitespace indentation instead of brackets and braces, YAML is the undisputed standard for cloud configuration files, CI/CD pipelines (GitHub Actions, GitLab CI), container orchestration (Kubernetes manifests), and Docker Compose. YAML 1.2 is a strict superset of JSON, meaning any valid JSON document is also valid YAML.

Click Blueprint to Enlarge

apiVersion: apps/v1
kind: Deployment
metadata:
  name: storage-gateway
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: telemetry/storage:v2.4
          resources: &standard_resources
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: "500m"
              memory: 1Gi

Created by GitHub co-founder Tom Preston-Werner, TOML aims to be a minimal configuration file format that is easy to read due to obvious semantics. Designed to map unambiguously to a hash table, TOML solves the human friction of YAML's significant indentation bugs and JSON's lack of comments. It is the dominant configuration standard in the Rust (Cargo.toml), Python (pyproject.toml), and Go ecosystems.

Click Blueprint to Enlarge

[package]
name = "vector-core"
version = "0.4.2"
edition = "2024"
authors = ["Data Infrastructure Team <[email protected]>"]

[dependencies]
tokio = { version = "1.38", features = ["full"] }
arrow = "52.0"
parquet = "52.0"

[database.connection]
host = "10.0.0.12"
port = 5432
timeout_seconds = 30.5
ssl_required = true

Protocol Buffers (Protobuf) is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. Developers write a schema in an Interface Definition Language (.proto), which a compiler (`protoc`) compiles into optimized native code across 15+ programming languages. Protobuf strips all field names from the binary payload, encoding fields as compact integer tags using variable-length zigzag integers (Varints). It is the backbone of gRPC, inter-microservice communication, and high-frequency systems across Google and enterprise architectures.

Click Blueprint to Enlarge

syntax = "proto3";

package analytics.v1;

message TelemetryEvent {
  string event_id = 1;
  uint64 timestamp_ns = 2;
  
  enum EventSeverity {
    INFO = 0;
    WARNING = 1;
    CRITICAL = 2;
  }
  
  EventSeverity severity = 3;
  map<string, string> metadata = 4;
  bytes raw_payload = 5;
}

Created by Google for performance-critical applications like game engines, Android graphics pipelines, and edge machine learning (TensorFlow Lite), FlatBuffers takes binary serialization to the extreme: zero-copy deserialization. Instead of allocating memory to unpack and parse bytes into language objects, FlatBuffers formats binary data on disk so that software can read fields directly out of raw memory or mmap() buffers via internal virtual table (vtable) offsets with zero allocation and zero parsing time.

Click Blueprint to Enlarge

namespace Rendering;

enum Color:byte { Red = 0, Green, Blue }

table Vec3 {
  x:float;
  y:float;
  z:float;
}

table Monster {
  pos:Vec3;
  mana:short = 150;
  hp:short = 100;
  name:string;
  inventory:[ubyte];
  color:Color = Blue;
}

root_type Monster;

MessagePack is an efficient binary serialization format that operates 'like JSON, but fast and small'. Unlike Protobuf and FlatBuffers, MessagePack does not require a prior schema file or compilation step; it is self-describing. By encoding JSON data structures into compact 1-byte headers (packing small integers into 1 byte and tagging short strings with minimum overhead), MessagePack provides a drop-in 2x to 5x performance and size improvement over JSON.

Click Blueprint to Enlarge

// JSON (27 bytes):
{"compact":true,"schema":0}

// MessagePack Hex (18 bytes):
// 82 (fixmap 2 items)
//   a7 63 6f 6d 70 61 63 74 (fixstr "compact")
//   c3 (true)
//   a6 73 63 68 65 6d 61 (fixstr "schema")
//   00 (fixint 0)

Developed by Hadoop co-founder Doug Cutting, Apache Avro is the premier row-oriented binary serialization system for large-scale data streaming and distributed processing. Avro stores the complete JSON schema directly inside the file header of every container file, meaning files are completely self-describing while data records remain ultra-compact without repeating field names or types. It is the gold standard for Apache Kafka message streams and Kafka Schema Registry.

Click Blueprint to Enlarge

{
  "type": "record",
  "name": "FinancialAuditRecord",
  "namespace": "com.bank.audit",
  "fields": [
    { "name": "tx_id", "type": "string" },
    { "name": "timestamp_epoch_ms", "type": "long" },
    { "name": "account_number", "type": "string" },
    { "name": "amount_cents", "type": "long" },
    { "name": "currency", "type": "string", "default": "USD" },
    { "name": "risk_score", "type": ["null", "float"], "default": null }
  ]
}

Apache Parquet is the undisputed king of modern analytical data formats. Born from Google's seminal 2010 Dremel paper and open-sourced jointly by Twitter and Cloudera, Parquet organizes tabular data by columns rather than rows. Instead of storing entire records contiguously, Parquet stores all values of column A together, then column B. This enables massive compression ratios (dictionary encoding, run-length encoding), projection pushdown (only reading queried columns), and predicate pushdown (skipping entire row groups using min/max statistics in the file footer). It powers modern data lakes, DuckDB, ClickHouse, Snowflake, Databricks, and Apache Spark.

Click Blueprint to Enlarge

[Magic: 'PAR1']
┌──────────────────────────────────────────────────────────┐
│ ROW GROUP 1 (e.g. 500,000 rows)                          │
│   ├── Column 'user_id'    [Dict Encoded, Snappy]         │
│   ├── Column 'timestamp'  [Delta Encoded, Bit-Packed]    │
│   └── Column 'amount'     [Plain Float, ZSTD]            │
├──────────────────────────────────────────────────────────┤
│ ROW GROUP 2 (e.g. 500,000 rows)                          │
│   └── ...                                                │
└──────────────────────────────────────────────────────────┘
[File Metadata Footer: Schemas, Col Stats (Min/Max), Offsets]
[Footer Length: 4 bytes]
[Magic: 'PAR1']

Optimized Row Columnar (ORC) is a high-performance columnar storage format conceived by Hortonworks and Apache Hive to maximize query speed and compression in Hadoop environments. Dividing files into self-contained 'Stripes' with integrated lightweight indexes (block-level Bloom filters and row group statistics every 10,000 rows), ORC is heavily optimized for Facebook Presto, Trino, and Apache Hive analytical engines.

Click Blueprint to Enlarge

┌──────────────────────────────────────────────────────────┐
│ STRIPE 1 (Default 64MB - 256MB)                          │
│   ├── Index Data: Min/Max stats + Bloom filters (per 10k)│
│   ├── Row Data: Columnar bit streams (compressed)        │
│   └── Stripe Footer: Encoding types, stream locations    │
├──────────────────────────────────────────────────────────┤
│ STRIPE 2 ...                                             │
├──────────────────────────────────────────────────────────┤
│ FILE FOOTER: Schema metadata, stripe statistics          │
│ POSTSCRIPT: Cache sizes, compression parameters, 'ORC'   │
└──────────────────────────────────────────────────────────┘

Apache Arrow Feather (now the official Arrow IPC file format) solves the data interchange bottleneck between modern data tools. In traditional computing, transferring data between Python (pandas), R, Rust (Polars), and C++ requires expensive serialization and deserialization. Arrow defines a standardized, language-independent columnar in-memory representation. The Feather format simply flushes this exact in-memory layout to disk. When read back, it can be mmap()'d into RAM instantly with zero copy and zero CPU parsing overhead.

Click Blueprint to Enlarge

/* 64-byte aligned SIMD in-memory chunk */
Length: 5 | Null Count: 1
Validity Bitmap: [1, 1, 0, 1, 1] (bit 2 is null)
Value Array:     [14, 28,  0, 42, 56] (int32 aligned)

Offsets Buffer:  [0, 4, 9, 14, 18, 24] (for variable strings)
Values Buffer:   "apple""grape""lemon""peach"
-> CPU reads directly with zero deserialization overhead!

Hierarchical Data Format 5 (HDF5) is designed to store and organize colossal quantities of numerical and scientific data. Think of HDF5 as an entire operating system file-system bundled into a single file: it contains 'Groups' (directories) and 'Datasets' (multidimensional N-dimensional arrays like tensors and matrices) with metadata attributes. Engineered for supercomputing, particle physics (CERN), climate modeling (NASA EarthData), and deep learning (early Keras weights), HDF5 handles petabyte datasets with hyperslab slicing.

Click Blueprint to Enlarge

import h5py
import numpy as np

# Create an HDF5 scientific container
with h5py.File("simulation_run.h5", "w") as f:
    # Groups behave like directories
    grp = f.create_group("atmospheric_model/2026")
    
    # Chunked, compressed 4D tensor (Time, Lat, Lon, Altitude)
    data = np.random.randn(365, 180, 360, 50)
    dset = grp.create_dataset(
        "pressure_tensor", 
        data=data, 
        chunks=(1, 180, 360, 10), 
        compression="gzip"
    )

# Selective hyperslab reading: Only 1 day, 1 altitude layer!
with h5py.File("simulation_run.h5", "r") as f:
    slice_day0 = f["atmospheric_model/2026/pressure_tensor"][0, :, :, 0]

Network Common Data Form (NetCDF) is the foundational data format for meteorology, oceanography, climatology, and Earth observation data. NetCDF provides self-describing, machine-independent array structures enriched with standard scientific metadata conventions (the Climate and Forecast / CF metadata conventions). Modern NetCDF-4 is built on top of HDF5, combining HDF5's high-speed chunked compression with standardized Earth science dimensions like latitude, longitude, and time.

Click Blueprint to Enlarge

netcdf global_sea_surface_temp {
dimensions:
    time = UNLIMITED ; // (365 currently)
    lat = 720 ;
    lon = 1440 ;
variables:
    float lat(lat) ;
        lat:units = "degrees_north" ;
    float lon(lon) ;
        lon:units = "degrees_east" ;
    double time(time) ;
        time:units = "days since 2026-01-01 00:00:00" ;
    float sst(time, lat, lon) ;
        sst:units = "kelvin" ;
        sst:_FillValue = -9999.0f ;
        sst:long_name = "Sea Surface Temperature" ;
}

Zarr represents the modern, cloud-native evolution of scientific array formats like HDF5 and NetCDF. While HDF5 relies on a single monolithic file with complex internal byte offsets that break over cloud object storage (S3 / GCS), Zarr splits massive multidimensional arrays into independent, compressed chunk files stored under predictable key-value paths (e.g., `/data/0.0.1`). Combined with lightweight JSON metadata, Zarr enables massively parallel reads and writes across thousands of serverless cloud functions without file locks.

Click Blueprint to Enlarge

// Key layout in Cloud Bucket:
// my-tensor.zarr/.zgroup
// my-tensor.zarr/.zattrs
// my-tensor.zarr/temperature/.zarray
// my-tensor.zarr/temperature/0.0.0  <- Binary compressed chunk
// my-tensor.zarr/temperature/0.0.1  <- Binary compressed chunk

// Contents of temperature/.zarray:
{
  "zarr_format": 2,
  "shape": [1000, 2048, 2048],
  "chunks": [100, 256, 256],
  "dtype": "<f4",
  "compressor": {
    "id": "blosc",
    "cname": "zstd",
    "clevel": 5
  }
}

GeoJSON is an open standard format designed for representing simple geographical features along with their non-spatial attributes. Based on JSON, it natively supports points, lines, polygons, and multi-part geometries using the World Geodetic System 1984 (WGS 84 / EPSG:4326) coordinate reference system. GeoJSON is supported natively by web mapping libraries like Leaflet, Mapbox GL, and OpenLayers, making it the easiest way to display geographic data in web browsers.

Click Blueprint to Enlarge

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "id": "boundary-curitiba",
      "properties": {
        "city": "Curitiba",
        "state": "Parana",
        "population": 1773733,
        "parks": 33
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [-49.2733, -25.4284],
            [-49.2600, -25.4300],
            [-49.2550, -25.4500],
            [-49.2800, -25.4450],
            [-49.2733, -25.4284]
          ]
        ]
      }
    }
  ]
}

GeoParquet brings the revolutionary columnar analytics power of Apache Parquet to geospatial vector data. Replacing legacy Shapefiles and bloated GeoJSON files, GeoParquet encodes geometries in Well-Known Binary (WKB) columns while storing spatial bounding boxes (bbox) in Parquet column statistics. This enables spatial predicate pushdown: analytical engines like DuckDB, Sedona, and BigQuery can query billions of geographic polygons and skip 99% of non-intersecting regions without reading them.

Click Blueprint to Enlarge

INSTALL spatial;
LOAD spatial;

-- Query 50 million buildings directly from cloud S3!
SELECT 
    id, 
    names.primary AS building_name,
    ST_Area(geometry) AS footprint_sq_meters
FROM read_parquet('s3://overturemaps-us-west-2/release/2026/buildings.parquet')
WHERE bbox.xmin >= -49.30 AND bbox.xmax <= -49.20
  AND bbox.ymin >= -25.50 AND bbox.ymax <= -25.40
LIMIT 10;

Cloud-Optimized GeoTIFF (COG) revolutionized satellite imagery and aerial photogrammetry by making raster files streaming-friendly over HTTP. Traditional TIFF files required downloading the entire multi-gigabyte file before rendering an overview. COG organizes pixels into internal tiles (e.g., 256x256 pixels) alongside pre-rendered multi-resolution pyramid overviews. Web clients and GIS servers use HTTP GET Range requests to stream only the exact pixels and zoom level needed for the current viewport.

Click Blueprint to Enlarge

┌──────────────────────────────────────────────────────────┐
│ TIFF Header & Offset Directory                           │
├──────────────────────────────────────────────────────────┤
│ OVERVIEW 16x (Thumbnail for global zoom)                │
│ OVERVIEW 4x  (Regional zoom tiles)                       │
│ FULL RESOLUTION (256x256 pixel tiles)                   │
│   ├── Tile (0,0)  |  Tile (0,1)  |  Tile (0,2)...        │
└──────────────────────────────────────────────────────────┘
Client performs:
GET /sentinel2_scene.tif HTTP/1.1
Range: bytes=4096-18432  --> Fetches ONLY the tile on screen!

JPEG is the most ubiquitous photographic image compression standard on Earth. Developed in 1992, JPEG leverages human visual psychoacoustics: the human eye is far more sensitive to brightness (luminance) than to fine color hue differences (chrominance). JPEG transforms 8x8 pixel blocks using the Discrete Cosine Transform (DCT), quantizes high-frequency details that the eye cannot see, and compresses the result using Huffman entropy coding. It enabled the explosion of digital photography and the early World Wide Web.

Click Blueprint to Enlarge

[RGB Image] 
     │
     ▼
[RGB -> YCbCr Color Conversion]
     │
     ▼
[Chroma Subsampling (4:2:0 / 4:2:2)]
     │
     ▼
[8x8 Pixel Block Splitting]
     │
     ▼
[Discrete Cosine Transform (DCT)]
     │
     ▼
[Quantization (Lossy Step: drops high frequencies)]
     │
     ▼
[Zig-Zag Scan & Huffman / Arithmetic Entropy Coding]
     │
     ▼
[.JPEG File (FF D8 FF E0...)]

Portable Network Graphics (PNG) was created in 1996 to replace the patented GIF format. PNG is an extensible chunk-based raster format providing bit-perfect lossless compression and full 8-bit / 16-bit alpha channel transparency (supporting 256 levels of smooth translucent anti-aliasing). By applying predictive 2D spatial filtering (Paeth, Sub, Up) before feeding scanlines to DEFLATE compression, PNG achieves high efficiency on screenshots, logos, diagrams, and digital graphics with sharp contrast.

Click Blueprint to Enlarge

[Magic: 89 50 4E 47 0D 0A 1A 0A]
┌──────────────────────────────────────────────────────────┐
│ IHDR Chunk: Width, Height, Bit depth, Color type, Filter │
├──────────────────────────────────────────────────────────┤
│ pHYs Chunk: Physical pixel dimensions (DPI)              │
├──────────────────────────────────────────────────────────┤
│ IDAT Chunk 1: [DEFLATE-compressed scanlines]             │
│ IDAT Chunk 2: [DEFLATE-compressed scanlines]             │
├──────────────────────────────────────────────────────────┤
│ IEND Chunk: File termination trailer                     │
└──────────────────────────────────────────────────────────┘

Developed by Google using technologies derived from the VP8 video codec, WebP was created to supersede both JPEG and PNG for the modern web. WebP supports lossy compression (typically 25-34% smaller than comparable JPEG images), lossless compression (26% smaller than PNG), alpha channel transparency in both modes, and animated sequences in a fraction of GIF's file size. Today, WebP is supported across 98%+ of global web browsers.

Click Blueprint to Enlarge

0x00: 'RIFF' (4 bytes)
0x04: File Size (4 bytes)
0x08: 'WEBP' (4 bytes)
0x0C: Chunk Header:
      - 'VP8 ' (Lossy VP8 video frame) OR
      - 'VP8L' (Lossless compressed bitmap) OR
      - 'VP8X' (Extended: Alpha channel, Animation, EXIF, ICC)
0x14: Data payload with entropy-coded macroblocks

The AV1 Image File Format (AVIF) represents the pinnacle of still-image compression. Standardized by the Alliance for Open Media (AOMedia), AVIF extracts the intra-frame encoding capabilities of the royalty-free AV1 video codec and wraps them in an ISO Base Media File Format (HEIF) container. AVIF achieves up to 50% smaller file sizes than JPEG and 20% smaller than WebP while introducing native 10-bit and 12-bit High Dynamic Range (HDR) color, wide color gamuts (BT.2020), and flawless alpha channels.

Click Blueprint to Enlarge

<!-- State-of-the-art web delivery tier -->
<picture>
  <!-- Next-gen tier: 50% smaller than JPEG -->
  <source srcset="/images/hero.avif" type="image/avif">
  
  <!-- Modern web tier: 30% smaller than JPEG -->
  <source srcset="/images/hero.webp" type="image/webp">
  
  <!-- Universal legacy fallback -->
  <img src="/images/hero.jpg" alt="Optimized visual asset" loading="lazy">
</picture>

Scalable Vector Graphics (SVG) is an XML-based markup format for describing two-dimensional vector graphics. Unlike raster bitmaps that store finite grids of pixels, SVG defines shapes mathematically using paths, Bézier curves, coordinates, and styling attributes. This means SVG graphics scale infinitely to any screen resolution, 8K monitor, or mobile display with crisp sharpness and zero pixelation. Because SVG is valid XML, it can be styled directly with CSS, animated with JavaScript, and manipulated inside the browser DOM.

Click Blueprint to Enlarge

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24">
  <style>
    .pulse-dot { fill: #10b981; animation: pulse 2s infinite; }
    @keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.3; } }
  </style>
  <circle cx="12" cy="12" r="10" stroke="#3b82f6" stroke-width="2" fill="none"/>
  <circle cx="12" cy="12" r="4" class="pulse-dot"/>
  <path d="M 12 2 L 12 6 M 12 18 L 12 22" stroke="#3b82f6" stroke-width="2"/>
</svg>

Concise Binary Object Representation (CBOR) is an IETF-standard binary data format designed specifically for constrained IoT devices and low-bandwidth networks (CoAP). Inspired by MessagePack but standardized with mathematical rigor by the IETF, CBOR uses an elegant 3-bit Major Type system supporting arbitrary-precision integers, raw byte strings, floating-point numbers, and semantic tags (e.g. tagging byte arrays as UUIDs or ISO datetimes). It powers WebAuthn (FIDO2 passkeys) and hardware security tokens worldwide.

Click Blueprint to Enlarge

Major Type 0: Unsigned Integer (0x00..0x17 inline, 0x18..0x1b byte follow)
Major Type 1: Negative Integer
Major Type 2: Byte String (Raw Binary)
Major Type 3: Text String (UTF-8)
Major Type 4: Array of data items
Major Type 5: Map of pairs (keys & values)
Major Type 6: Semantic Tag (e.g., Tag 1 = Epoch timestamp)
Major Type 7: Float & Simple values (false, true, null)

BSON (Binary JSON) is the binary storage and wire format powering MongoDB and document-oriented databases. Designed to be lightweight, traversable, and efficient, BSON extends JSON's limited type system by adding first-class support for 64-bit integers (int64), exact Decimals (Decimal128), Datetimes (UTC epoch ms), raw binary buffers, Regular Expressions, and 12-byte ObjectIDs. Its length-prefixed fields allow database query engines to skip unqueried subdocuments with pointer arithmetic without parsing their interior.

Click Blueprint to Enlarge

document ::= int32 e_list "\x00"  // 4-byte total length + elements + null terminator
element  ::= "\x01" e_name double           // 64-bit float
           | "\x02" e_name string           // UTF-8 string
           | "\x03" e_name document         // Embedded sub-document
           | "\x04" e_name document         // Array (keys: "0", "1", "2")
           | "\x05" e_name binary           // Raw binary bytes
           | "\x07" e_name (byte*12)        // 12-byte ObjectID
           | "\x09" e_name int64            // UTC Datetime (ms since epoch)
           | "\x10" e_name int32            // 32-bit Integer

Created by Kenton Varda (the primary author of Google's Protocol Buffers v2), Cap'n Proto is designed around a radical premise: 'Infinity times faster than Protocol Buffers'. Cap'n Proto eliminates the encode/decode step altogether. The bytes that software builds in memory are identical to the bytes written to disk or sent across the network. With word-aligned structs and direct pointer offsets, deserialization takes 0.0 seconds of CPU time. It also features a time-traveling RPC protocol based on Promise Pipelining.

Click Blueprint to Enlarge

@0xdbb9ad1f14bf0b36;

struct NetworkPacket {
  id @0 :UInt64;
  timestamp @1 :Int64;
  srcIp @2 :Text;
  dstIp @3 :Text;
  
  payload :union {
    rawBytes @4 :Data;
    httpPayload @5 :HttpMetadata;
  }
}

struct HttpMetadata {
  method @0 :Text;
  statusCode @1 :UInt16;
  headers @2 :List(HeaderPair);
}

struct HeaderPair {
  key @0 :Text;
  value @1 :Text;
}

Abstract Syntax Notation One (ASN.1) is one of the most foundational and battle-tested data specification systems in computer history. Standardized in 1984 by international telecom bodies (ITU-T), ASN.1 separates data structure definitions from their physical byte encodings (DER, BER, PER). It is the indispensable cryptographic backbone of the global Internet: every single HTTPS TLS/SSL certificate, RSA public key, x509 credential, cellular 4G/5G radio signaling protocol, and air traffic control system is encoded in ASN.1 DER.

Click Blueprint to Enlarge

// ASN.1 Module Definition
Certificate ::= SEQUENCE {
    tbsCertificate       TBSCertificate,
    signatureAlgorithm   AlgorithmIdentifier,
    signatureValue       BIT STRING
}

// DER Hex Byte Breakdown:
// 30 82 04 2A -> Tag 0x30 (SEQUENCE), Length 0x042A (1066 bytes)
//   30 82 03 12 -> Sub-Sequence: TBSCertificate
//     A0 03 02 01 02 -> Context Tag 0, Version 3 (v3 certificate)
//     02 10 4F 3A... -> Tag 0x02 (INTEGER), Serial Number (16 bytes)

Apache Iceberg is a high-performance open table format for huge analytic datasets. Conceived at Netflix to solve the concurrency and performance failures of legacy Hive metastores, Iceberg manages tables as an immutable snapshot tree. Instead of tracking directories on object storage, Iceberg tracks exact file lists via Avro manifest files. This unlocks full ACID transactions, atomic partition evolution without rewriting data, time-travel queries (querying data as it existed last week), and simultaneous multi-engine concurrency across Spark, Trino, Flink, and DuckDB.

Click Blueprint to Enlarge

table_location/
├── metadata/
│   ├── v3.metadata.json           <-- Current table state (schema, partition spec)
│   ├── snap-9041-manifest-list.avro <-- Points to all active manifests for Snapshot 9041
│   └── 4a8b-manifest.avro         <-- Lists active Parquet files + column min/max stats
└── data/
    ├── event_date=2026-10-10/
    │   ├── 00001-42-data.parquet  <-- Physical columnar records
    │   └── 00002-43-data.parquet

Delta Lake is an open-source storage layer that brings ACID transactions and reliability to Apache Spark and big data workloads. Invented by Databricks, Delta Lake manages tables through an ordered, append-only JSON transaction log (`_delta_log/000000.json`). Every commit is an atomic write to the log. Periodically, log commits are compacted into Parquet checkpoint files. Delta Lake features schema enforcement (rejecting bad data writes), streaming and batch unification, and time travel.

Click Blueprint to Enlarge

// Example: _delta_log/00000000000000000042.json
{
  "commitInfo": {
    "timestamp": 1791686400000,
    "operation": "WRITE",
    "operationParameters": { "mode": "Append" }
  }
}
{
  "add": {
    "path": "part-00001-c000.snappy.parquet",
    "size": 1845920,
    "modificationTime": 1791686400000,
    "dataChange": true,
    "stats": "{\"numRecords\": 50000, \"minValues\": {\"id\": 1}, \"maxValues\": {\"id\": 50000}}"
  }
}

The SQLite database file format is recognized by the US Library of Congress as a recommended storage format for the preservation of digital content. Far more than a simple file format, SQLite packs an entire ACID-compliant relational SQL engine into a single cross-platform disk file. Structured around fixed-size pages (default 4096 bytes) organized into B-Tree data structures with a Write-Ahead Log (WAL), SQLite files operate identically on big-endian and little-endian machines without data conversion.

Click Blueprint to Enlarge

Offset 0..15:  "SQLite format 3\x00" (Magic Header String)
Offset 16..17: Database page size (e.g. 0x1000 = 4096 bytes)
Offset 18..19: File format write/read version (1 = rollback, 2 = WAL)
Offset 24..27: File change counter (increments on every commit)
Offset 40..43: Schema cookie (increments on DDL ALTER/CREATE)
Offset 56..59: Text encoding (1 = UTF-8, 2 = UTF-16le, 3 = UTF-16be)
Offset 100+:   Page 1 data begins (Root of sqlite_schema table)

The Portable Document Format (PDF) is the universal international standard (ISO 32000) for fixed-layout digital documents. Conceived by Adobe co-founder John Warnock's 'Camelot Project' in 1991, PDF preserves identical visual presentation, typography, vector drawings, raster imagery, and layout across any computer screen, printer, and operating system. Built upon PostScript's imaging model, PDF organizes text, fonts, and graphics into an indirect object graph indexed by a random-access cross-reference (`xref`) table at the end of the file.

Click Blueprint to Enlarge

%PDF-1.7 (Header)
1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj
2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj
3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R >> endobj
4 0 obj << /Length 52 >>
stream
BT /F1 24 Tf 100 700 Td (Hello World, PDF!) Tj ET
endstream
endobj
xref
0 5
0000000000 65535 f 
0000000009 00000 n 
trailer << /Size 5 /Root 1 0 R >>
startxref
280
%%EOF

MPEG-4 Part 14 (MP4) is the universal digital multimedia container format of the modern era. Derived directly from Apple's QuickTime file format (`.mov`), MP4 encapsulates multiple video, audio, subtitle, and chapter tracks into a structured hierarchy of 4-byte tagged 'atoms' or 'boxes'. It is supported natively by virtually every television, browser, smartphone, social media platform, and GPU hardware video decoder on the planet.

Click Blueprint to Enlarge

[ftyp] File Type Box (e.g. major_brand = 'isom')
[moov] Movie Metadata Container Box
  ├── [mvhd] Movie Header: duration, timescale, volume
  ├── [trak] Track 1: Video (H.264 / AV1)
  │     ├── [tkhd] Track Header
  │     └── [mdia] -> [minf] -> [stbl] (Sample Table: frame offsets, keyframes)
  └── [trak] Track 2: Audio (AAC / Opus)
[mdat] Media Data Box: Interleaved raw audio & video packets

MPEG-1 Audio Layer III (MP3) revolutionized human culture and the global music industry by making digital audio portable over the Internet. Engineered by Germany's Fraunhofer Institute, MP3 exploits the psychoacoustics of human hearing: when a loud sound occurs, the human ear cannot detect quieter sounds at nearby frequencies (auditory masking). MP3 strips out these inaudible audio components and compresses the remaining frequencies using Modified Discrete Cosine Transform (MDCT) and Huffman coding, shrinking CD audio tracks by 90% with minimal perceived quality loss.

Click Blueprint to Enlarge

[ID3v2 Header: Metadata, Album Art, Artist]
[Audio Frame 1]
  ├── Frame Sync (11 bits: 11111111 111...)
  ├── MPEG Version: MPEG-1
  ├── Layer: Layer III
  ├── Bitrate: 320 kbps
  ├── Sampling Rate: 44.1 kHz (CD quality)
  └── Side Information & Huffman-coded subband samples
[Audio Frame 2] ...
[Audio Frame N] ...

Free Lossless Audio Codec (FLAC) is the open-source gold standard for bit-perfect audio preservation. Unlike lossy formats (MP3, AAC) that permanently discard audio frequencies, FLAC compresses audio waveforms without losing a single bit of information: decompressing a FLAC file reproduces the original master studio WAV or CD audio track byte-for-byte identically. Using linear predictive coding to model continuous audio waveforms followed by Golomb-Rice entropy coding, FLAC reduces file sizes by roughly 50% while supporting high-resolution 24-bit / 192kHz studio master audio.

Click Blueprint to Enlarge

[Magic: 'fLaC']
[METADATA_BLOCK_STREAMINFO]
  ├── Min/Max Block Size (e.g. 4096 samples)
  ├── Sample Rate: 96,000 Hz (Hi-Res Audio)
  ├── Channels: 2 (Stereo)
  ├── Bits per Sample: 24 bits
  └── Total Samples & MD5 Audio Signature (for integrity verification)
[METADATA_BLOCK_VORBIS_COMMENT: ARTIST=..., ALBUM=...]
[METADATA_BLOCK_PICTURE: Hi-Res Cover Art]
[FRAME 1: Sync word -> LPC Subframes -> Rice encoded residual bytes]
[FRAME 2: ...]

WebAssembly (WASM) is a binary code format and virtual machine designed to execute code at near-native speed inside web browsers and serverless runtimes. Compiling languages like C++, Rust, Go, and C# into a compact, structured binary representation, WASM allows complex database engines (SQLite in browser, DuckDB-WASM, PGlite), game engines (Unreal, Unity), and video processing pipelines (FFmpeg-WASM) to run securely inside the browser sandbox with isolated linear memory.

Click Blueprint to Enlarge

0x00: 00 61 73 6d (Magic: \0asm)
0x04: 01 00 00 00 (Version 1)
Section 1 (Type Section): Function signatures [i32, i32] -> [i32]
Section 2 (Import Section): Imports from host environment
Section 3 (Function Section): Function declarations
Section 5 (Memory Section): Linear memory limits (1 page = 64KB)
Section 7 (Export Section): Exported function names
Section 10 (Code Section): Function bodies (Stack instructions: local.get, i32.add)

GeoPackage (GPKG) is the modern, open-standard replacement for the obsolete 1990s ESRI Shapefile. Standardized by the Open Geospatial Consortium (OGC), GeoPackage is built directly on top of an SQLite database file. It packages vector features (points, lines, polygons), raster map tile pyramids, non-spatial attribute tables, and Coordinate Reference Systems (CRS) into a single, compact, cross-platform file without the file-splitting bugs and 2GB limits of legacy GIS formats.

Click Blueprint to Enlarge

-- GeoPackage Core Metadata Tables (inside SQLite):
SELECT * FROM gpkg_spatial_ref_sys;  -- Defines Coordinate Systems (EPSG:4326, EPSG:3857)
SELECT * FROM gpkg_contents;         -- Catalogs all vector layers and tile matrices
SELECT * FROM gpkg_geometry_columns; -- Identifies geometry columns and spatial types

-- Query vector features directly with SQL:
SELECT 
    id, 
    municipality_name, 
    population, 
    geom 
FROM territorial_boundaries 
WHERE population > 100000;

Tagged Image File Format (TIFF) is the granddaddy of professional raster formats. Created in 1986 by Aldus and Adobe for desktop publishing and scanners, TIFF is an extensible tagged container that can store uncompressed or losslessly compressed raster bitmaps with arbitrary bit depths (8, 16, 32-bit float per channel), multi-page image stacks, and multi-spectral satellite channels. It serves as the physical container for GeoTIFF and Cloud-Optimized GeoTIFF (COG).

Click Blueprint to Enlarge

Offset 0..1: Byte Order ('II' = 0x4949 Little Endian, 'MM' = 0x4D4D Big Endian)
Offset 2..3: Magic number 42 (0x002A)
Offset 4..7: Offset to the first Image File Directory (IFD)

[Image File Directory (IFD)]
  ├── Tag 256 (0x0100): ImageWidth = 4096
  ├── Tag 257 (0x0101): ImageLength = 2160
  ├── Tag 258 (0x0102): BitsPerSample = 16, 16, 16 (48-bit RGB)
  ├── Tag 259 (0x0103): Compression = 5 (LZW)
  └── Tag 273 (0x0111): StripOffsets -> Pointers to raw pixel strips

The Bitmap image format (BMP), or Device-Independent Bitmap (DIB), was introduced by Microsoft and IBM in early versions of Windows and OS/2. BMP stores raster images in the simplest, most primitive way possible: a brief header followed by raw, uncompressed pixel color values stored line by line (conventionally ordered from bottom to top). Because it performs virtually zero compression, BMP files are enormous, but require zero CPU horsepower to decode.

Click Blueprint to Enlarge

[BITMAPFILEHEADER - 14 bytes]
  0x00: "BM" (0x42 0x4D - Magic signature)
  0x02: Total file size in bytes
  0x0A: Pixel data offset (typically 54 bytes)

[BITMAPINFOHEADER - 40 bytes]
  0x0E: Header size (40)
  0x12: Image Width (signed integer)
  0x16: Image Height (signed integer, positive = bottom-up)
  0x1A: Color Planes (1)
  0x1C: Bits per pixel (24 = RGB, 32 = RGBA)
  0x1E: Compression method (0 = BI_RGB uncompressed)

[Raw Pixel Array]
  Row 1 (Bottom line): B G R [pad to 4-byte boundary]
  Row 2: ...

glTF (GL Transmission Format), heralded as the 'JPEG of 3D', is an open, royalty-free specification for real-time delivery of 3D models and scenes. Engineered specifically to minimize runtime processing on client devices, glTF combines a JSON scene graph with tightly-packed contiguous binary buffers (.bin or packed inside a binary .glb container). These buffers can be uploaded directly into GPU VRAM (via WebGL, WebGPU, Vulkan, or Metal) without CPU restructuring.

Click Blueprint to Enlarge

[0x00..0x03]: 0x46546C67 (Magic bytes 'glTF')
[0x04..0x07]: 0x00000002 (Version: 2)
[0x08..0x0B]: Total File Length in Bytes (uint32)
--- Chunk 0: JSON Manifest (Nodes, Meshes, Materials) ---
[0x0C..0x0F]: Chunk Length
[0x10..0x13]: 0x4E4F534A ('JSON')
--- Chunk 1: Binary Buffer (Contiguous GPU Vertex Arrays) ---
[Chunk 1 Len]: 0x004E4942 ('BIN\0')

Universal Scene Description (OpenUSD) is the collaborative backbone of modern Hollywood VFX, animated feature films, and industrial digital twins. Developed by Pixar and standardized through the Alliance for OpenUSD (Pixar, Apple, NVIDIA, Adobe, Autodesk), USD allows hundreds of artists to simultaneously collaborate on massive, complex virtual worlds using non-destructive LIVRPS composition rules (Local, Inherits, VariantSets, References, Payloads, Specializes).

Click Blueprint to Enlarge

#usda 1.0
(
    defaultPrim = "HeroVehicle"
    metersPerUnit = 0.01
    upAxis = "Y"
)

def Xform "HeroVehicle" (
    prepend references = @./assets/vehicle_body.usdc@
)
{
    float3 xformOp:translate = (0, 150, 0)
    uniform token[] xformOpOrder = ["xformOp:translate"]
    
    over "Body" (
        prepend variantSets = "paintStyle"
    ) {
        variantSet "paintStyle" = {
            "MetallicRed" {
                color3f[] primvars:displayColor = [(0.9, 0.05, 0.05)]
            }
        }
    }
}

SafeTensors is the secure, high-performance successor to Python's pickle-based checkpoint formats (.pt, .bin, .ckpt). Created by Hugging Face, SafeTensors completely eliminates remote code execution (RCE) vulnerabilities by restricting the file to pure numeric tensor weights. It combines an 8-byte header size prefix and a JSON metadata table with contiguous binary buffers, enabling zero-copy OS memory mapping directly into GPU memory in milliseconds.

Click Blueprint to Enlarge

[0x00..0x07]: uint64 Little-Endian (Header Size N, e.g. 0x0000000000000210 = 528 bytes)
[0x08..0x08+N]: UTF-8 JSON Header String:
{
  "__metadata__": {"format": "pt"},
  "transformer.layers.0.attn.q_proj.weight": {
    "dtype": "BF16",
    "shape": [4096, 4096],
    "data_offsets": [0, 33554432]
  }
}
[0x08+N..EOF]: Raw Contiguous Tensor Memory Buffer (Directly mmaped into GPU)

GGUF (GPT-Generated Unified Format) is the universal file format powering local, private LLM execution on consumer laptops and edge devices. Engineered by the llama.cpp community, GGUF bundles everything required to run an LLM—quantized weights, hyperparameter limits, BPE tokenizer vocabularies, and chat prompt templates—into a single self-contained file. It revolutionized local AI by allowing 70-billion-parameter models to run on MacBook unified memory and consumer CPUs.

Click Blueprint to Enlarge

[0x00..0x03]: 0x46554747 ('GGUF')
[0x04..0x07]: uint32 Version (3)
[0x08..0x0F]: uint64 Tensor Count (e.g. 291)
[0x10..0x17]: uint64 Metadata KV Count (e.g. 36)
--- Extensible Key-Value Metadata Entries ---
- "general.architecture"        -> STRING: "llama"
- "llama.context_length"        -> UINT32: 8192
- "tokenizer.ggml.tokens"       -> ARRAY[STRING]: ["<unk>", "<s>", "hello", ...]
--- Tensor Information Directory & Aligned Quantized Weights ---
- "blk.0.attn_q.weight"         -> GGML_TYPE_Q4_K_M [4096, 4096]

DICOM is the bedrock of modern medical diagnostics, deployed across every hospital, imaging clinic, and radiology department on Earth. Beyond merely storing images, DICOM is an all-encompassing information protocol binding high-bit-depth pixel data (CT, MRI, ultrasound, X-ray) with calibrated clinical metadata, physical 3D patient slice geometry, radiation dosage logs, and hospital administrative records.

Click Blueprint to Enlarge

[0x00..0x7F]: 128-byte Preamble (Usually zero-filled)
[0x80..0x83]: 0x4449434D ('DICM' Magic Bytes)
--- Sequential Tagged Data Elements (Group, Element) ---
Tag (0010, 0010): VR=PN Len=14 Value="DOE^JOHN^^^^" (Patient Name)
Tag (0020, 000D): VR=UI Len=42 Value="1.2.840.113619.2.55..." (Study UID)
Tag (0028, 0010): VR=US Len=02 Value=512 (Rows)
Tag (0028, 0011): VR=US Len=02 Value=512 (Columns)
Tag (0028, 0100): VR=US Len=02 Value=16  (Bits Allocated = 16-bit Grayscale)
Tag (7FE0, 0010): VR=OW Len=524288 Value=[Raw CT Hounsfield Unit Sensor Pixels]

FITS is the definitive data format of modern observational astronomy and astrophysics. Adopted under the legendary charter 'Once FITS, Always FITS', every major observatory—including the James Webb Space Telescope (JWST), Hubble Space Telescope, and ALMA radio array—archives its raw and processed observations in FITS. Files consist of 2880-byte blocks starting with human-readable ASCII keyword cards followed by raw multidimensional arrays of IEEE 754 floating point pixels.

Click Blueprint to Enlarge

SIMPLE  =                    T / file conforms to FITS standard
BITPIX  =                  -32 / IEEE 32-bit floating point pixels
NAXIS   =                    2 / 2-dimensional image array
NAXIS1  =                 2048 / Fast axis pixel length
NAXIS2  =                 2048 / Slow axis pixel length
TELESCOP= 'JWST                ' / James Webb Space Telescope
INSTRUME= 'NIRCAM              ' / Near-Infrared Camera
CRVAL1  =        180.423750000 / Reference celestial RA (degrees)
CRVAL2  =        -18.864320000 / Reference celestial Dec (degrees)
BUNIT   = 'MJy/sr              ' / Physical flux units
END
[... 2880-byte padded boundary followed by 2048x2048 raw IEEE-754 floats ...]

BAM (Binary Alignment Map), CRAM, and VCF (Variant Call Format) represent the technological computational backbone of modern genomics, precision medicine, and oncology. Rather than standard monolithic compression, genomic files utilize BGZF (Blocked GNU Zip), which splits streams into 64KB independently compressed blocks. Combined with companion index files (.bai or .tbi), bioinformaticians can query specific chromosome coordinates in milliseconds without reading multi-gigabyte files.

Click Blueprint to Enlarge

# VCF Variant Call Format (BGZF Compressed):
#CHROM  POS       ID          REF  ALT  QUAL  FILTER  INFO
chr1    14370     rs6054257   G    A    29    PASS    AF=0.5;DP=14;NS=3
chr1    17330     .           T    A    3     q10     AF=0.017;DP=11
# Binary BAM Sequence Alignment Record:
[refID: 0][pos: 14370][bin_mq_nl: 0x4B][flag_nc: 0x63][l_seq: 150]
[CIGAR String: 150M (150 Matches)]
[Sequence: 4-bit encoded nucleotide bases]
[Phred Quality Scores: ASCII offset 33]

GPX and KML are the universal languages of personal navigation, fitness telemetry, and 3D globe visualization. GPX (GPS Exchange Format) is the global standard for GPS fitness watches, bike computers, and outdoor devices (Garmin, Strava, Apple Watch), recording time-ordered trackpoints with elevation and biometric sensors. KML (Keyhole Markup Language), originally developed for Keyhole and standardized by the Open Geospatial Consortium (OGC), specializes in rich 3D geographic styling, camera perspectives, and polygon rendering for Google Earth.

Click Blueprint to Enlarge

<?xml version="1.0" encoding="UTF-8"?>
<gpx version="1.1" creator="Garmin Edge 1040">
  <trk>
    <name>Chamonix Alpine Trail Climb</name>
    <trkseg>
      <trkpt lat="45.923700" lon="6.869400">
        <ele>1035.4</ele>
        <time>2026-06-15T06:30:00Z</time>
        <extensions>
          <gpxtpx:TrackPointExtension>
            <gpxtpx:hr>145</gpxtpx:hr>
            <gpxtpx:cad>86</gpxtpx:cad>
          </gpxtpx:TrackPointExtension>
        </extensions>
      </trkpt>
    </trkseg>
  </trk>
</gpx>

TAR and ZIP form the invisible foundation of computing, software deployment, and operating systems. TAR (Tape Archive) concatenates files sequentially into 512-byte blocks preserving POSIX file ownership and permissions, typically paired with gzip or zstd. ZIP revolutionized PC computing by placing a 'Central Directory' at the end of the archive, enabling random-access single-file extraction without decompressing the entire file. Modern application formats (DOCX, XLSX, APK, JAR) are internally standard ZIP files.

Click Blueprint to Enlarge

[0x00..Local File Header 1]: 0x04034B50 ('PK\x03\x04') [Compressed File Data 1]
[Offset..Local File Header 2]: 0x04034B50 ('PK\x03\x04') [Compressed File Data 2]
...
--- Central Directory (Located at Archive Tail) ---
[Central Directory Header 1]: 0x02014B50 -> Points to Local Header 1 offset
[Central Directory Header 2]: 0x02014B50 -> Points to Local Header 2 offset
--- End of Central Directory Record (EOCD) ---
[0x06054B50 ('PK\x05\x06')]
- uint16 Total Entries in Central Directory
- uint32 Size of Central Directory
- uint32 Offset of Central Directory starting position

EPUB is the universal open standard for digital e-books, adopted across Apple Books, Kobo, Google Play Books, and Amazon Kindle. Rather than imposing static paper pagination like PDF, EPUB packages an entire responsive, accessible website inside a ZIP archive. Text reflows dynamically across smartphone screens, e-ink displays, and desktop monitors according to user font size and screen geometry.

Click Blueprint to Enlarge

book.epub (ZIP Container):
├── mimetype (Uncompressed: 'application/epub+zip' at offset 38)
├── META-INF/
│   └── container.xml (Points to root package document)
└── OEBPS/
    ├── content.opf   (Metadata, Manifest of items, Reading Spine order)
    ├── toc.xhtml     (Navigation Table of Contents)
    ├── styles/
    │   └── typography.css
    ├── text/
    │   ├── chapter01.xhtml
    │   └── chapter02.xhtml
    └── images/
        └── cover.jpg