Unit Normalization in Vendor Feeds: Currency, Temperature, and Dates

The founding insight for HyperNorm came from a specific incident: temperature sensor readings from two equipment monitoring vendors, one outputting Celsius and the other outputting Fahrenheit. Both were ingested into the same column declared as Celsius in the warehouse. For about three months, an anomaly detection model trained on that column was producing wrong alerts for readings from the Fahrenheit vendor. A value like 77F (a normal ambient temperature) was being interpreted as 77C, which is well above the equipment's normal operating range.

Nobody noticed for three months because the pipeline ran without errors and the alerts were plausible: the equipment vendor had a history of occasional high-temperature readings. The systematic bias in one direction for that vendor's readings was visible in retrospect but invisible in the moment. This post covers how to prevent this class of problem in a multi-vendor data pipeline.

Why Unit Errors Are Structurally Different from Type Errors

A type error is detectable. A string in an integer column raises an exception. A 64-bit float where you expected 32-bit causes precision loss that can be measured. Type errors have a mechanical signature in the pipeline.

Unit errors have no mechanical signature. A temperature value of 77 is valid regardless of whether it is Celsius or Fahrenheit. A revenue value of 1200 is valid regardless of whether it is USD, EUR, or INR. The value passes every type check, every range check (assuming the range is set in the wrong unit), and every null check. The error only becomes visible when you compare values across sources and notice the systematic offset, or when a downstream model produces predictions that diverge from reality.

This is why unit declaration needs to be a first-class attribute in your data schema, not a comment in a documentation file that no pipeline actually reads.

Currency: The Most Common Silent Unit Error

Currency mismatch is the highest-frequency unit error in multi-vendor B2B data. The scenario is straightforward: Vendor A reports revenue in USD because they serve primarily US customers. Vendor B reports revenue in EUR because they're based in Germany. Vendor C has inconsistent currency per record because their export format changed midway through their own migration.

A canonical schema that declares revenue: unit: usd needs to know the source unit for each vendor field to apply the right conversion. This requires either:

1. A unit declaration in the vendor connector config, or
2. A currency code field in each vendor record that the pipeline can read at runtime

The second approach is more reliable when available. Many vendor APIs include a currency code field alongside monetary amounts. If your pipeline ignores it and just ingests the numeric value, you're discarding information you need.

canonical_fields:
  revenue:
    type: decimal
    unit: usd
    conversion_rules:
      - detect_from_field: currency_code
        method: fx_rate
        fx_source: ecb_daily
      - static_source_unit: eur
        applies_to: [vendor_b_connector]
        method: fx_rate
        fx_source: ecb_daily

The detect_from_field rule is preferred when the vendor sends a currency code. The static_source_unit rule is a fallback for vendors that always send one currency without declaring it in a field. Mixing the two approaches in a single connector config is a code smell: it means part of the vendor data is self-describing and part is not, which typically indicates a schema change happened at some point in the vendor's history and both approaches coexist in your data.

Temperature: The Formula Problem

Temperature conversion is not a simple multiplier. The formula from Fahrenheit to Celsius is (F - 32) * 5/9, which means a pipeline that tries to handle temperature conversion with a unit ratio (as it would for, say, meters to feet) will produce wrong answers.

The conversion needs to be declared as a named formula, not as a linear scaling factor:

canonical_fields:
  temperature_ambient:
    type: decimal
    unit: celsius
    conversion_rules:
      - source_unit: fahrenheit
        method: formula
        formula: "(value - 32) * 5 / 9"
      - source_unit: kelvin
        method: formula
        formula: "value - 273.15"

Temperature Kelvin appears in scientific equipment data. If your pipeline only handles Celsius and Fahrenheit, you'll get large negative Celsius values from Kelvin sources near room temperature (around 298K becomes ~25C; a value of 50K would produce -223C, which is physically impossible for most equipment and should trigger a range validation error rather than a silent wrong value).

Date and Timestamp Normalization

Date format variation is the unit normalization problem most teams encounter first, because it's the most visually obvious: a date that looks like 03/04/2025 is ambiguous between March 4 and April 3, and a date that looks like 2025-14-03 will fail ISO 8601 parsing entirely.

The normalization approach for dates:

1. Declare the source format per vendor connector
2. Parse to a canonical datetime object using the declared format
3. Store in UTC ISO 8601 in the canonical output
4. Fail loudly on parse error rather than producing NULL

from datetime import datetime, timezone

def normalize_date(
    raw: str,
    source_format: str,
    source_tz_offset: int = 0
) -> str:
    dt = datetime.strptime(raw, source_format)
    dt = dt.replace(tzinfo=timezone.utc)
    return dt.isoformat()

Timezone handling deserves explicit attention. A vendor that sends timestamps without a timezone indicator but operates in UTC+5:30 (IST) is implicitly sending IST times. If your pipeline treats them as UTC, you have a 5.5-hour offset error in every timestamp. For most analytical use cases this is not critical; for real-time anomaly detection or event ordering it is.

Declare the assumed timezone offset in the connector config and apply it at parse time. Do not rely on the vendor to include timezone information in the timestamp string, even when their documentation says they do. In practice, timezone handling in vendor APIs is inconsistent, and the safe assumption is that you need to declare it yourself.

Measurement Units: Surface Area and Volume

Physical measurement unit variation comes up in logistics, manufacturing, and equipment monitoring contexts. Common patterns: one vendor sends package dimensions in centimeters and weight in kilograms, another sends inches and pounds. A third vendor sends volumes in gallons where your canonical schema expects liters.

These conversions are linear (unlike temperature) and can be handled with a simple multiplier declared in the connector config:

canonical_fields:
  package_weight_kg:
    type: decimal
    unit: kg
    conversion_rules:
      - source_unit: lb
        method: multiply
        factor: 0.453592
  package_volume_liters:
    type: decimal
    unit: liters
    conversion_rules:
      - source_unit: gallons_us
        method: multiply
        factor: 3.78541
      - source_unit: gallons_uk
        method: multiply
        factor: 4.54609

Note the distinction between US and UK gallons. This is a real difference (about 20%) that will produce wrong inventory calculations if you treat them as the same unit. The US gallon is the default assumption in North American logistics systems; UK gallon appears in older systems and some European integrations. Confirm which variant your vendor uses before declaring the conversion factor.

Range Validation as a Unit Error Detector

Declaring known valid ranges for physical measurements provides a secondary detection layer for unit errors that slip through config gaps. If your canonical temperature_ambient field has a valid range of -40C to 150C (appropriate for most industrial equipment monitoring), a value of 350 in that field is either an equipment malfunction or a Fahrenheit value that wasn't converted. Either way, it should go to the anomaly log rather than the clean canonical output.

Range validation is not a substitute for unit declaration. It catches egregious mismatches but misses plausible ones: a temperature of 30C is a plausible ambient temperature in Celsius AND a plausible ambient temperature in Fahrenheit (86F, converted to 30C). Range validation does not catch this specific case. But it does catch the 77F-as-77C scenario that affected the original pipeline: 77C is at the high end of most industrial equipment's normal operating range and would trigger a review.

Build range validation as a complement to unit declaration, not a replacement for it. You need both.