# Recording

Video of the product being used — multi-page flows, real interactions, WebGL captured as
painted, with an optional deterministic mode whose output does not depend on machine speed.

```bash
npm run record:dry   -- recordings/studio-tour/script.json  # validate, resolve selectors
npm run record       -- recordings/studio-tour/script.json  # realtime capture
npm run record:exact -- recordings/studio-tour/script.json  # deterministic capture
npm run record       -- <script> --formats mp4,gif,vertical
npm run record       -- <script> --live https://joeasare.com
npm run record       -- <script> --keep-frames              # keep frames for inspection
```

Set `"capture": "deterministic"` in the script to make it the default for that recording.

`npm run record:test` validates every script in `recordings/` and runs inside `npm run gates`.

## Why a script file rather than browser code

A recording is `recordings/<name>/script.json`. It is declarative because more than one
agent touches it: a director writes the shots and their intent, an interaction developer
resolves the selectors and timings, a critic judges the frames against the stated intent.
A number in a JSON file can be diffed, reviewed and adjusted; a hundred lines of
imperative Playwright cannot be critiqued usefully by anyone who did not write it.

`name` must be a lowercase slug matching its directory. **Every shot needs an `intent`** —
the validator rejects a shot without one, because it is what a critique is measured
against.

## Architecture

| file | job |
|---|---|
| `packages/reel/src/cli.mjs` | entry point; validate, capture, encode, write `capture.json` |
| `packages/reel/src/runner.mjs` | owns the browser for the whole story; launch flags; one context |
| `packages/reel/src/actions.mjs` | the verb vocabulary, all eased |
| `packages/reel/src/capture.mjs` | CDP screencast to timestamped frames |
| `packages/reel/src/assemble.mjs` | ffmpeg: master MP4, 9:16 crop, GIF, critic samples |
| `packages/reel/src/cursor.mjs` | synthetic pointer, injected before page scripts |
| `packages/reel/src/clock.mjs` | the virtual clock shim for deterministic capture |
| `packages/reel/src/timeline.mjs` | the two timing drivers behind one `wait`/`animate` seam |
| `packages/reel/src/critic.mjs` | judges a finished recording from its frames |
| `packages/reel/src/edit.mjs` | edit decision lists — resolve segments against a capture |
| `packages/reel/src/edit-cli.mjs` | `npm run record:edit` |
| `packages/reel/src/caption.mjs` | captions rendered in Chromium, in studio typography |
| `packages/reel/src/frame-grid.mjs` | downsampled luminance grids for frame comparison |
| `packages/reel/src/schema.mjs` | validation and the dry run |
| `packages/reel/src/server.mjs` | static server honouring `cleanUrls`, with commerce endpoints stubbed |

**One browser context for the entire story.** That is what makes a multi-page narrative
work: cookies, `localStorage` and in-page state survive navigation, so a later shot can
show a state an earlier shot created. A recorder that opens a page per shot can only
produce disconnected clips.

**The cursor is drawn, not real.** The OS pointer is not part of what the compositor
paints, so a screencast of a click shows the effect and never the cause — fields appear to
fill themselves. `cursor.mjs` injects a pointer that follows real `pointermove` events and
draws a ripple on click. It is `pointer-events:none` and tagged `data-recording-chrome`, so
it cannot alter what it documents.

**Frames carry real timestamps.** The screencast delivers a frame when the compositor
paints one, which is irregular — a heavy Three.js scene may give 8fps for a second and
30fps the next. Every frame's true timestamp goes into an ffmpeg concat manifest and the
encode resamples to a constant rate. Treating irregular frames as evenly spaced is what
makes naive WebGL captures appear to speed up and slow down.

## Two capture modes

### `realtime` (default)

The compositor paints when it can and every frame carries a real timestamp. Fast to run —
roughly wall-clock — but the output is bound to machine speed:

| content | measured motion fps |
|---|---|
| DOM pages, scrolling and typing | 20–23 |
| Three.js under SwiftShader | ~10 |

No launch flag lifts this. `--disable-frame-rate-limit` and `--disable-gpu-vsync` were
measured and made it **worse** (15.9fps), so they are deliberately absent. Dropping JPEG
quality from 92 to 35 bought about 3fps, which is why the default sits at 80.

There is a second, larger problem that is easy to miss: **realtime output does not match
the duration the script asks for.** Driver round trips, paint waits and per-keystroke IPC
all add to every action, and they compound. The four-shot `studio-tour` declares about 21
seconds and produced **87.5 seconds** of video. A director writing a 30-second social clip
was getting two minutes.

### `deterministic`

Page time is replaced by a virtual clock (`clock.mjs`, installed before any page script
runs). The recorder advances it exactly one frame, runs the timers and animation callbacks
that became due, screenshots, and repeats.

| | realtime | deterministic |
|---|---|---|
| frame rate | 10–23, varies | exactly `fps` |
| duration vs script | ~4x over | matches |
| same script twice | different | **byte-identical** |
| wall-clock cost | ~1x | ~6x |

Measured on `studio-tour`: 515 frames at exactly 24fps, 21.5s of video from a script
declaring ~21s, 152ms of real work per frame, and two runs producing identical frame
checksums. The 3D shot has 50 unique frames out of 50 — the render loop is genuinely being
driven, not frozen.

**Why a JS shim rather than CDP `Emulation.setVirtualTimePolicy`:** virtual time freezes
the whole renderer, so a page awaiting a real network response hangs until the policy is
nursed through pending fetches. The shim replaces only the *timing* APIs, so network,
decoding and layout continue on real time and a page that fetches mid-recording simply
works.

**What it cannot virtualise:** `<video>`/`<audio>` playback; CSS animations are *sampled*
rather than stepped (paused, with `currentTime` set each frame — exact for time-based
animations, wrong for anything keyed to `transitionend` timing); `new Date()`, whose
constructor is deliberately left alone because replacing it breaks date maths in harder-to-
see ways; and worker or iframe timers, which have their own global scopes.

**Reproducibility has a sharp edge.** Anything non-deterministic in an action breaks it.
Keystroke jitter is seeded (`resetJitter`, xorshift32) and the gate fails on `Math.random`
or a direct `page.waitForTimeout` anywhere in `actions.mjs` — either would silently opt out
of the virtual clock while still looking correct in realtime mode.

### Which to use

Realtime for iterating on a script: it runs at 1x and you are looking at composition, not
smoothness. Deterministic for anything anyone else will see.

## The critic

Every recording is judged from its own frames and writes `critique.json` beside the video.
No model call — it runs on pixels and the capture timeline, so it works with no egress and
gives the same answer twice.

```
✓ critique 100/100 — 0 finding(s)
```

| axis | weight | catches |
|---|---|---|
| `deadFrames` | 0.30 | a shot that rendered nothing — blank or a uniform wash |
| `frozenMotion` | 0.30 | a shot claiming movement whose frames never change |
| `continuity` | 0.15 | an isolated jump inside a shot, where the eye reads a cut |
| `pacing` | 0.15 | a shot too short for anyone to read |
| `ending` | 0.10 | a clip that lands mid-motion instead of on a held state |

`frozenMotion` is the one worth having. A WebGL scene that never animated produces frames
that each look perfectly correct, so a reviewer flicking through stills cannot see it —
only comparing consecutive frames can. Findings carry a timecode and the shot id, and a
frozen-shot finding quotes the shot's stated intent back, which is what the required
`intent` field is for.

Add `--gate` to make the verdict decide the exit code, for a loop that should stop on a
bad take rather than quietly producing one.

**The plan for this phase was wrong and was corrected.** It said to point `design-eval`'s
still-image rubric at the samples. That rubric's axes are `subjectFirst`, `framing`,
`geometryCredibility` and `materials` — product-photography measures. A frame of someone
filling in a form has no subject on a background and no material response, so those axes
would have scored noise and dressed it in a calibrated-looking number. What carried over
is the *shape*: measured axes, blockers separated from warnings, and a `skip` status that
renormalises rather than scoring a default.

### Two things it got wrong before it got them right

**Comparing luminance aggregates could not see movement.** The first version diffed
`imageStats`'s mean, spread and occupancy. A pure translation leaves all three unchanged,
so a scrolling repeated pattern — or a 3D object rotating under constant light — measured
as perfectly still, and the critic reported moving shots as frozen. It now compares 16x16
downsampled luminance grids (`frame-grid.mjs`), which see position. Coarse on purpose:
a pixel-exact comparison would read compression noise as constant motion.

**Large change is not the same as discontinuity.** Scrolling the landing page from its
dark hero into the cream section below produces enormous frame-to-frame deltas, and every
one was reported as a cut. A jump is *isolated* — one large change among small ones —
so it must also stand clear of its local median. Separately, the analysis strip is coarser
than a shot boundary, so the delta straddling a `goto` was being filed under the shot it
landed in; both ends of a shot are now treated as legitimate cuts.

Both survive as regression cases in `tools/test-record-critic.mjs`, alongside a fixture
for every axis. That file lives apart from `test-record.mjs` because it needs a pixel
backend, which CI installs only for the `tests` job — and it **fails rather than skips**
when the backend is missing, since a silently skipped gate is how one stops gating.

## Edits — cutting a short piece from a master

A social clip is not a different recording, it is a different *reading* of one. Cutting
from the master keeps the two consistent, costs seconds rather than minutes, and means the
30-second version cannot drift from the two-minute version it claims to summarise.

```bash
npm run record:edit -- recordings/studio-tour/social.edit.json --check   # resolve only
npm run record:edit -- recordings/studio-tour/social.edit.json
```

```json
{
  "name": "studio-tour-social",
  "source": "studio-tour",
  "aspect": "9:16",
  "budgetSec": 30,
  "segments": [
    { "shot": "start", "from": 0.6, "to": 6.5, "fit": "contain",
      "kicker": "Brand it live", "caption": "Your palette, applied as you type." },
    { "shot": "webgl", "from": 2.0, "focusX": 0.62,
      "kicker": "Real WebGL", "caption": "Live 3D — not a video of it." }
  ]
}
```

Segments address the master **by shot id**, resolved against the timeline in
`capture.json`, so an edit survives the script being re-timed — shots move, the edit still
points at the right material. `from`/`to` alongside a shot are offsets *within* it, which
is how a director thinks about trimming: "the last two seconds of the orbit", not "second
19.6 of the file". An edit naming a shot that no longer exists is refused rather than cut
from zero.

| field | |
|---|---|
| `aspect` | `16:9` · `9:16` · `1:1` |
| `fit` | `cover` crops to fill (default) · `contain` letterboxes on the studio ground |
| `focusX` | 0–1, where a `cover` crop sits. Per segment, falling back to the edit |
| `speed` | 0.25–4, retimes a segment |
| `budgetSec` | reported when exceeded, never enforced |
| `caption` · `kicker` | burned in, in the studio's own typefaces |

`budgetSec` advises rather than trimming: silently retiming a cut to hit a number changes
the pacing the director chose, and pacing is the whole point of a short format.

### Captions are rendered by Chromium

The obvious route is ffmpeg's `drawtext`. It is not available — the bundled `ffmpeg-static`
reports `--enable-libfreetype` but ships **no `drawtext` filter**, leaving only libass,
which renders in whatever face fontconfig finds on the machine. A caption that is one
typeface locally and another in CI is not a caption, it is a defect that appears only in
the deliverable.

So captions are rendered by the browser the recorder already launches, using the studio's
own `tokens.css`, and overlaid as transparent PNGs. Real typeface, real kerning, matching
the pages being recorded — with no font vendored into the repo.

### Framing a landscape master vertically

A 1440x900 master narrowed to 9:16 keeps only **35% of its width**. That frames a centred
3D stage well and slices the left column off a two-column form. Hence `fit`:

- `cover` where a *detail* is the subject — a 3D stage, a single control.
- `contain` where the *layout* is the subject — a form, a dense page. It letterboxes on the
  studio background, and the resulting empty band is where the caption sits.

For a vertical piece that is mostly dense UI, the better answer is to record a vertical
take rather than crop a landscape one — `setViewport`, or a script with a vertical
`viewport`.

## Roadmap

**Phase 1 — done.** Multi-page flows with preserved state, real interactions, live page
manipulation, WebGL captured as painted, MP4/GIF/vertical output, a schema gate and a
dry run.

**Phase 2 — done.** Deterministic capture, above. Both modes share one timing seam
(`timeline.mjs`): actions ask the driver to `wait` or to `animate`, and the driver decides
whether that means sleeping or advancing a virtual frame. That is what let phase 2 land
without rewriting the verb vocabulary.

**Phase 3 — done.** The frame critic, below.

**Phase 4 — done.** Edits, below.

## Retired

`tools/inject-recorder-all.mjs` injected a canvas-only recorder button into the blueprint
editors. It emitted emoji, which `tools/verify.mjs` treats as an **error**, so running it
broke the build; and `canvas.captureStream()` sees only the 3D surface, so the DOM and all
text overlays were invisible in the result. It now exits with a pointer here.

`assets/lib/recorder.js` is still loaded by the blueprint editors and was not removed — it
does a genuinely different job, letting a customer export their own 3D scene from the
browser.