> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbased.health/llms.txt
> Use this file to discover all available pages before exploring further.

# Data pipeline

> How getbased turns stored profile data into active chart, scoring, and AI data.

# Data Pipeline

`getActiveData()` in `js/data.ts` is the central active-data projection. Dashboard, category charts, compare, correlations, scoring, and AI context consume its output. It reads `state`, caches by relevant input identities and profile context, and returns a derived dataset without persisting changes.

## Pipeline flowchart

```
MARKER_SCHEMA + customMarkers
              │
              ▼
sex-specific defaults + OPTIMAL_RANGES + refOverrides
              │
              ▼
entries[] → sorted dates + merged same-date values + entryContextByDate
              │
              ▼
values[] projection and single-point category projection
              │
              ▼
recorded/eligible predicted cycle phases → phaseRefRanges[]
              │
              ▼
age/sex/time/fasting context → contextRefRanges[] / contextOptimalRanges[]
              │
              ▼
direct lab calculations or deterministic calculated fallbacks
              │
              ▼
optional display-unit conversion → marker placement projection → cached data
```

The caller is responsible for applying date range filtering after receiving `data`:

```js theme={null}
const data = getActiveData();
const filtered = filterDatesByRange(data); // returns a filtered projection
```

## Marker identity and storage keys

Persisted values and companion maps continue to use `"category.markerKey"` storage keys:

```
diabetes.insulin
hormones.testosterone
lipids.triglycerides
hematology.wbc
```

This storage format is used in:

* `UNIT_CONVERSIONS` keys
* `OPTIMAL_RANGES` keys
* `PHASE_RANGES` keys
* `importedData.entries[].markers` — each entry stores its values keyed this way
* `importedData.customMarkers` — custom marker definitions
* AI prompt references in `buildMarkerReference()`

It is not the marker's immutable identity or necessarily its rendered category. Built-in markers use stable `gb:marker:*` IDs, custom definitions carry `custom:*` IDs, and `importedData.markerPlacements` can select a different visual category. After the pipeline finishes its native calculations and unit conversion, `applyMarkerPlacements()` projects accepted placements and attaches `markerId`, `storageDotKey`, `nativeCategoryKey`, and `displayCategoryKey` to active markers.

Mutation code must use `marker.storageDotKey`; it must not reconstruct a persisted key from `displayCategoryKey` or a rendered card ID. See [Marker model and category placement](/developers/marker-model).

Alias migration resolves historical dotKeys before projection. See [Lab markers and range internals](/developers/lab-markers-ranges-internals#schema-ownership) for the range and migration contracts.

## Entry storage format

`importedData.entries` is the live per-date lab dataset. Per-file provenance lives separately in `importSnapshots[]`. Each entry has a date, a flat markers object, and optional per-draw context:

```js theme={null}
{
  date: "2025-03-15",
  context: {
    sampleTime: "07:35",
    fasting: true,
    cycleDay: 21,
    cyclePhase: "luteal",
    cyclePhaseDetail: "mid_luteal",
    cyclePhaseSource: "recorded"
  },
  collectionContextSources: {
    sampleTime: "snap_...",
    fasting: "snap_..."
  },
  markers: {
    "biochemistry.glucose": 5.2,
    "hormones.testosterone": 18.4,
    "lipids.triglycerides": 1.1,
    // ...
  }
}
```

Multiple entries on the same date are merged into one value lookup and one normalized `entryContextByDate` record. Snapshot-aware commit helpers own conflict and restoration behavior; do not bypass them when mutating imported data.

## Values arrays — aligned with dates

Every marker in the output has a `values` array aligned with `data.dates`:

```js theme={null}
data.dates   = ["2024-09-01", "2024-12-01", "2025-03-15"]
marker.values = [4.8,          5.1,          null         ]
//                                           ^ no result for this date
```

`null` means the marker was not measured on that date. Charts use `spanGaps: true` to draw lines across gaps. Status functions check for `null` before evaluating.

## singlePoint categories

The `fattyAcids` category has `singlePoint: true` in the schema. These markers typically come from a single test, not a time series. The pipeline handles them differently:

* Only the **latest** entry date is used, stored as `cat.singleDate`
* Each marker gets `marker.singlePoint = true` and a single-element `marker.values` array
* Views render grid cards instead of trend charts for these categories

## Custom markers

Markers not in `MARKER_SCHEMA` are auto-imported from PDFs. The pipeline merges them into `data.categories` at runtime:

```js theme={null}
// importedData.customMarkers:
{
  "mylab.cortisol": {
    markerId: "custom:opaque_id",
    name: "Cortisol (AM)",
    unit: "nmol/L",
    refMin: 170,
    refMax: 720,
    categoryLabel: "My Lab"
  }
}
```

If the category key does not exist in the schema, a new category is created with a bookmark icon. Custom markers have `marker.custom = true` and are treated identically to schema markers in all views. Legacy custom definitions receive deterministic stable IDs without re-keying their stored values.

## Contextual ranges and calculated markers

The projection attaches date-aligned phase, reference, and optimal range arrays after values are populated. Imported or manual reference overrides block generic context fallbacks, while eligible phase-specific hormone ranges remain per-date. Unknown required context yields no range or an explicit unrated label.

Calculated values are computed in-pipeline in the native `calculatedRatios` category. A profile may then display a calculated marker in a compatible regular category, but calculation and storage resolution still happen at its native key.

`populateCalculatedMarkers()` prefers a direct supported lab result already present in `calculatedRatios` for that date and computes only missing destinations. This is required for FIB-4, anion gap, and lab-reported ratios whose rounding or method can differ from the fallback formula.

PhenoAge and Bortz Age are hidden calculated components. Biological Age combines both when available and falls back to either one. Explicit creatinine-unreliable low-muscle context gates the calculation.

See [Lab markers and range internals](/developers/lab-markers-ranges-internals) for range precedence, all calculation formulas, collection-context ownership, phase gates, and alias migration requirements.

## Durable import and restore boundary

`import-commit-validation.ts` rejects duplicate canonical mappings and blank/non-finite values before mutation. Compatible custom values and range bounds convert together; raw per-file snapshots retain source units. Specialty marker identities stay separate.

IndexedDB writes commit on transaction completion, not merely request success. Reads propagate failure, and failed marker edits/JSON imports roll back rather than leaving partially applied state. Restore merges entries by marker and retains unrelated same-date results, collection-context ownership, timestamps and deletion metadata.

Import snapshots are historical evidence; entries are the live projection. Preserve source links through edits. Repair an older missing link only when one matching snapshot establishes provenance; do not guess between multiple files.

## The `data` parameter pattern

The first `getActiveData()` call for a changed input set clones and processes the schema; matching calls reuse the active-data cache. Views should still accept an optional `data` parameter so one projection can feed a complete render and so dependencies remain explicit:

```js theme={null}
// Good: one pipeline call, passed to all sub-renders
function refreshAll() {
  const data = getActiveData();
  const filtered = filterDatesByRange(data);
  showDashboard(filtered);
}

// Bad: each sub-render calls getActiveData() independently
function refreshAll() {
  showDashboard();        // calls getActiveData() internally
  showCategory(cat);     // calls getActiveData() again
}
```

When writing a rendering function, accept `data` when its caller already owns the projection. Call `invalidateActiveDataCache()` from mutation paths that change a tracked input; storage save paths already do this.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.