What Actually Happens Inside a Normalization Pipeline

Most teams treat normalization as a single step. You call some transform function, get clean data back, move on. In practice, normalization is four distinct operations that run in a specific order, and each one has its own class of failure mode. This post walks through each stage as HyperNorm actually implements it, including what happens when things go wrong at each step.

Before the Pipeline Runs: Schema Inference

Schema inference is a bootstrapping step that runs on the first ingestion from a new vendor source. HyperNorm samples a configurable window of records (default: 10,000 rows or 5 MB, whichever comes first) and derives a typed schema. The output is a canonical field map stored as a versioned JSON artifact:

{
  "canonical_fields": {
    "revenue_usd": {
      "vendor_aliases": ["gross_rev", "REVENUE_USD", "Rev_usd"],
      "inferred_type": "decimal",
      "source_types_seen": ["string", "float64", "integer"],
      "unit": "usd"
    }
  }
}

The alias map carries the weight going forward. When a vendor changes a field name from gross_rev to gross_revenue, the new name gets appended to vendor_aliases on the next successful run. This is not magic: it requires either automated schema drift detection triggering an alias update, or manual config maintenance. We'll come back to what happens when neither fires in time.

Most normalization implementations treat schema inference as a one-time step. That assumption breaks on any vendor that ships API updates. The schema artifact needs a version history, not a single mutable record.

Stage 1: Field Name Canonicalization

Once you have a schema artifact, field canonicalization is string normalization against a lookup table. The mechanical steps run in a fixed order:

1. Strip leading and trailing whitespace
2. Fold to lowercase
3. Convert camelCase to snake_case
4. Remove vendor-specific prefixes (configured per connector)
5. Match against the alias map

import re

def canonicalize_field_name(raw_name: str, alias_map: dict) -> str | None:
    name = raw_name.strip().lower()
    name = re.sub(r'([a-z])([A-Z])', r'\1_\2', name).lower()
    name = name.lstrip('vnd_').lstrip('src_')
    return alias_map.get(name)

The None return on no match is intentional. Fields that don't match any alias route to an unmatched-field log rather than being silently dropped. Silent drops are how unit mismatches go undetected for months. An unmatched field in the log at least tells you something changed.

Field name variation across vendors for the same concept is more common than most teams expect before they've seen it in production. A single concept like "company revenue" can appear as gross_revenue_usd, total_rev, RevAmount, and revenue_eur across four vendors. The canonicalization step needs all of those aliases registered to handle all four correctly.

Stage 2: Type Coercion

After field mapping, every value for a given canonical field is coerced to the declared type. This is where a large number of pipelines silently produce NULLs or truncation errors without raising any alarm.

The coercion rules HyperNorm applies by default:

Source type	Target type	Rule
string "1,234.56"	decimal	Strip commas, parse float
string "EUR1.234,56"	decimal	Strip symbol, swap separators, parse float
string "true"/"false"	boolean	Case-insensitive parse
integer	decimal	Widening cast, no precision loss
string "2023-14-03"	date	Flag as invalid, emit to error log

The last row is the important one. Invalid date strings don't produce a NULL in the output. They produce an entry in the coercion error log with the source row ID, the field name, the raw value, and the expected pattern. A pipeline run that suddenly produces 3,000 date coercion errors is a signal that the vendor changed their date format. Emitting NULLs instead would hide that signal entirely.

Stage 3: Unit Conversion

Unit conversion is the most opaque stage because the failures it prevents are numerically plausible. A revenue field in EUR that gets treated as USD doesn't produce a pipeline error. It produces wrong numbers that pass all type validation.

HyperNorm's approach is declaration-first. The canonical schema for a field includes a unit attribute. If the incoming vendor data declares a different unit in its metadata, or if the field name pattern suggests a different unit (for example, a _eur suffix), the pipeline applies a conversion before writing to canonical output.

canonical_fields:
  revenue:
    unit: usd
    conversion_rules:
      - source_unit: eur
        method: fx_rate
        fx_source: ecb_daily
      - source_unit: gbp
        method: fx_rate
        fx_source: ecb_daily

The fx_source: ecb_daily setting tells the pipeline to use the European Central Bank's published daily reference rate. This rate is fetched once per pipeline run and cached for the run duration. This is not tick-level precision, and it is not meant to be. The goal is to eliminate the orders-of-magnitude error that comes from treating EUR amounts as USD, not to handle intraday FX fluctuations.

Temperature and measurement unit conversion use the same declaration pattern. The pipeline needs to know the source unit, not just the canonical target, to apply the right conversion factor.

Stage 4: Null Handling and Sentinel Detection

Null handling deserves its own stage because vendors encode missing values in at least six different ways: actual JSON null, the string "null", the string "N/A", the string "-", empty string "", and the integer 0 (sometimes meaning "no data" in older export formats).

HyperNorm's sentinel detection runs before type coercion. You configure a per-source sentinel list:

{
  "sentinels": {
    "vendor_karbon": ["null", "N/A", "-", ""]
  }
}

Any value that matches a sentinel is replaced with a proper NULL before coercion runs. This prevents "N/A" from failing type coercion and being emitted as a coercion error when it was an intentional null from the start.

The integer 0 sentinel needs careful handling. Revenue of 0 is a valid value in many contexts. Configure 0 as a sentinel only at the field level, not the source level, and only for fields where zero is genuinely not a valid business value (foreign keys, status codes).

What the Output Record Looks Like

After all four stages, the output is a canonical record conforming to your declared schema. HyperNorm emits this as Parquet by default, with JSON and CSV available for downstream compatibility. Each record carries a provenance header:

{
  "_hnai_source": "vendor_karbon",
  "_hnai_run_id": "run_2025121901",
  "_hnai_schema_version": "v3",
  "_hnai_coercion_errors": 0,
  "_hnai_fields_unmatched": 1
}

The _hnai_fields_unmatched: 1 in that header is something your monitoring should catch. One unmatched field on a first run is typical. One unmatched field appearing after three weeks of a stable connector run means the vendor added or renamed something. The alert on that metric has caught more schema changes for our early customers than any schema fingerprinting check.

What the Normalization Stage Does Not Do

This is worth being explicit about. The normalization stage does not deduplicate records. It does not resolve entity identity. A normalized record containing "Acme Corp" in the company_name field and another containing "Acme Corporation" both pass through the normalization pipeline as valid records. They will be clean and consistently typed when they arrive at the entity resolution stage.

Normalization makes data structurally consistent. Entity identity is a separate problem that runs on top of structurally consistent data. Running entity resolution on unnormalized data is a real failure mode: the match scores are lower because the same entity has differently typed or formatted fields across sources, and your false-negative rate climbs.

Config Drift Is a Pipeline Risk Too

One thing we didn't fully anticipate when building this: the normalization config itself can drift out of sync with the vendor schema over time, in the same way the vendor schema drifts. If you add a new canonical field but don't update the alias map for an existing connector, that field produces NULLs for that connector's records until someone investigates.

HyperNorm tracks config coverage as a metric: what percentage of a vendor's inbound fields are matched by the current alias map. A coverage drop from 97% to 84% on a vendor run signals a config gap, not just a data quality issue. The distinction matters because the fix is different: schema drift requires an alias map update; data quality issues require upstream investigation with the vendor.