Data Pipeline

getActiveData() in js/data.js is the central active-data projection. Dashboard, category charts, compare, correlations, scoring, and AI context consume its output. It reads state, caches by relevant input identities and profile context, and returns a derived dataset without persisting changes.

Pipeline flowchart

MARKER_SCHEMA + customMarkers
              │
              ▼
sex-specific defaults + OPTIMAL_RANGES + refOverrides
              │
              ▼
entries[] → sorted dates + merged same-date values + entryContextByDate
              │
              ▼
values[] projection and single-point category projection
              │
              ▼
recorded/eligible predicted cycle phases → phaseRefRanges[]
              │
              ▼
age/sex/time/fasting context → contextRefRanges[] / contextOptimalRanges[]
              │
              ▼
direct lab calculations or deterministic calculated fallbacks
              │
              ▼
optional display-unit conversion → marker placement projection → cached data

The caller is responsible for applying date range filtering after receiving data:

const data = getActiveData();
const filtered = filterDatesByRange(data); // returns a filtered projection

Marker identity and storage keys

Persisted values and companion maps continue to use "category.markerKey" storage keys:

diabetes.insulin
hormones.testosterone
lipids.triglycerides
hematology.wbc

This storage format is used in:

UNIT_CONVERSIONS keys
OPTIMAL_RANGES keys
PHASE_RANGES keys
importedData.entries[].markers — each entry stores its values keyed this way
importedData.customMarkers — custom marker definitions
AI prompt references in buildMarkerReference()

It is not the marker’s immutable identity or necessarily its rendered category. Built-in markers use stable gb:marker:* IDs, custom definitions carry custom:* IDs, and importedData.markerPlacements can select a different visual category. After the pipeline finishes its native calculations and unit conversion, applyMarkerPlacements() projects accepted placements and attaches markerId, storageDotKey, nativeCategoryKey, and displayCategoryKey to active markers. Mutation code must use marker.storageDotKey; it must not reconstruct a persisted key from displayCategoryKey or a rendered card ID. See Marker model and category placement. Alias migration resolves historical dotKeys before projection. See Lab markers and range internals for the range and migration contracts.

Entry storage format

importedData.entries is the live per-date lab dataset. Per-file provenance lives separately in importSnapshots[]. Each entry has a date, a flat markers object, and optional per-draw context:

{
  date: "2025-03-15",
  context: {
    sampleTime: "07:35",
    fasting: true,
    cycleDay: 21,
    cyclePhase: "luteal",
    cyclePhaseDetail: "mid_luteal",
    cyclePhaseSource: "recorded"
  },
  collectionContextSources: {
    sampleTime: "snap_...",
    fasting: "snap_..."
  },
  markers: {
    "biochemistry.glucose": 5.2,
    "hormones.testosterone": 18.4,
    "lipids.triglycerides": 1.1,
    // ...
  }
}

Multiple entries on the same date are merged into one value lookup and one normalized entryContextByDate record. Snapshot-aware commit helpers own conflict and restoration behavior; do not bypass them when mutating imported data.

Values arrays — aligned with dates

Every marker in the output has a values array aligned with data.dates:

data.dates   = ["2024-09-01", "2024-12-01", "2025-03-15"]
marker.values = [4.8,          5.1,          null         ]
//                                           ^ no result for this date

null means the marker was not measured on that date. Charts use spanGaps: true to draw lines across gaps. Status functions check for null before evaluating.

singlePoint categories

The fattyAcids category has singlePoint: true in the schema. These markers typically come from a single test, not a time series. The pipeline handles them differently:

Only the latest entry date is used, stored as cat.singleDate
Each marker gets marker.singlePoint = true and a single-element marker.values array
Views render grid cards instead of trend charts for these categories

Custom markers

Markers not in MARKER_SCHEMA are auto-imported from PDFs. The pipeline merges them into data.categories at runtime:

// importedData.customMarkers:
{
  "mylab.cortisol": {
    markerId: "custom:opaque_id",
    name: "Cortisol (AM)",
    unit: "nmol/L",
    refMin: 170,
    refMax: 720,
    categoryLabel: "My Lab"
  }
}

If the category key does not exist in the schema, a new category is created with a bookmark icon. Custom markers have marker.custom = true and are treated identically to schema markers in all views. Legacy custom definitions receive deterministic stable IDs without re-keying their stored values.

Contextual ranges and calculated markers

The projection attaches date-aligned phase, reference, and optimal range arrays after values are populated. Imported or manual reference overrides block generic context fallbacks, while eligible phase-specific hormone ranges remain per-date. Unknown required context yields no range or an explicit unrated label. Calculated values are computed in-pipeline in the native calculatedRatios category. A profile may then display a calculated marker in a compatible regular category, but calculation and storage resolution still happen at its native key. populateCalculatedMarkers() prefers a direct supported lab result already present in calculatedRatios for that date and computes only missing destinations. This is required for FIB-4, anion gap, and lab-reported ratios whose rounding or method can differ from the fallback formula. PhenoAge and Bortz Age are hidden calculated components. Biological Age combines both when available and falls back to either one. Explicit creatinine-unreliable low-muscle context gates the calculation. See Lab markers and range internals for range precedence, all calculation formulas, collection-context ownership, phase gates, and alias migration requirements.

The `data` parameter pattern

The first getActiveData() call for a changed input set clones and processes the schema; matching calls reuse the active-data cache. Views should still accept an optional data parameter so one projection can feed a complete render and so dependencies remain explicit:

// Good: one pipeline call, passed to all sub-renders
function refreshAll() {
  const data = getActiveData();
  const filtered = filterDatesByRange(data);
  showDashboard(filtered);
}

// Bad: each sub-render calls getActiveData() independently
function refreshAll() {
  showDashboard();        // calls getActiveData() internally
  showCategory(cat);     // calls getActiveData() again
}

When writing a rendering function, accept data when its caller already owns the projection. Call invalidateActiveDataCache() from mutation paths that change a tracked input; storage save paths already do this.

Core project

Data model & sync

AI & agents

Feature internals

Security architecture

Operations & release

Data pipeline

Data Pipeline

Pipeline flowchart

Marker identity and storage keys

Entry storage format

Values arrays — aligned with dates

singlePoint categories

Custom markers

Contextual ranges and calculated markers

The `data` parameter pattern

​Data Pipeline

​Pipeline flowchart

​Marker identity and storage keys

​Entry storage format

​Values arrays — aligned with dates

​singlePoint categories

​Custom markers

​Contextual ranges and calculated markers

​The data parameter pattern

Data Pipeline

Pipeline flowchart

Marker identity and storage keys

Entry storage format

Values arrays — aligned with dates

singlePoint categories

Custom markers

Contextual ranges and calculated markers

The `data` parameter pattern