engine-3d

Source: engine-3d.js:26

3D biometric pipeline for Ghostati.

Parallel to engine.js (face-api / 128-D / euclidean distance), this module implements the ImageEmbedder path:

• feature embedding extracted from a canvas or video element
• Cosine similarity as the distance metric (higher = more similar)
• Separate localStorage DB (state.db3d) sharing IDs with the 2D DB
• MATCH_THRESHOLD_3D: similarity >= threshold counts as a match

IMPORTANT — what this is NOT: The embedder was trained for generic image similarity, not face recognition. It responds strongly to global visual context (background, lighting, colour palette) as much as to facial geometry. False positives on similar backgrounds are expected and are part of the didactic point: this is a "feature-level" engine, not a biometric one.

This module does NOT dispatch matchStateChanged. All events are dispatched by the orchestrator in main.js / auto-find-loop.js after composing the unified payload from both engines.

Instance Methods

hasActivePlugin3d(): boolean

Helper: is any 2D or 3D plugin currently active? Mirrors hasActivePlugin() from engine.js without importing it.

Returns

• boolean

Static Methods

loadMobileNet(embedderInstance: object | null): Promise.<void>

Load the 3D embedder and store it on state. Safe to call multiple times — returns immediately if already loaded.

Historical note: the exported name stays loadMobileNet() to preserve the public API used by main.js, even though the implementation now loads MediaPipe ImageEmbedder with MobileNetV3 Small.

Parameters

• embedderInstance (object | null) — Optional injected embedder for tests.

Returns

• Promise.<void>

• See:
  • main.js – called during init after face-api models are loaded.

cosineSimilarity( vecA: Array.<number>, vecB: Array.<number>, ): number

Cosine similarity between two numeric vectors. Returns a value in [-1, 1]; 1 = identical direction. Higher values mean MORE similar (opposite sign convention from L2 distance).

Parameters

• vecA (Array.<number>)
• vecB (Array.<number>)

Returns

• number

getFaceEmbedding( source: HTMLCanvasElement | HTMLVideoElement, ): Promise.<Array.<number>>

Extract an ImageEmbedder embedding from a canvas or video element.

NOTE: ImageEmbedder responds to global image content (background, colours, texture) as much as to the face itself. This is a known characteristic and is the didactic subject of the workshop.

Parameters

• source (HTMLCanvasElement | HTMLVideoElement)

Returns

• Promise.<Array.<number>> — Embedding vector serialisable to JSON.

seekFaceInDb3d(embedding: Array.<number>): Object

Scan the 3D DB for the best cosine-similarity match against embedding.

Parameters

• embedding (Array.<number>) — Query vector.

Returns

• Object

saveFace3d( id: number, ): Promise.<({id: number, liveInfo3d: object}|null)>

Save an ImageEmbedder embedding to state.db3d under the given id. The id is assigned by engine.js (saveFace) and passed here so both DBs stay in sync without a shared counter.

Parameters

• id (number) — ID already assigned by the 2D engine's saveFace.

Returns

• Promise.<({id: number, liveInfo3d: object}|null)> — Returns null if model not ready or DB not initialised.

compositeAndDetect3d( baseCanvas?: HTMLCanvasElement | null, ): Promise.<({canvas: HTMLCanvasElement, embedding: Array.<number>}|null)>

Build a composited canvas and extract an ImageEmbedder embedding from it.

The base of the composite is, in order of preference:

1. baseCanvas — the 2D-composited frame from engine.js's compositeAndDetect (video + active 2D Ghostyle already baked in). This is what makes a 2D overlay such as the Face Brush actually move the 3D similarity score: the embedder must see the painted pixels, not the bare video. Without it the ImageEmbedder embeds the un-obfuscated frame and reports a match no matter how much the user paints.
2. the raw video frame, when no 2D composite was supplied (e.g. only a 3D/UV plugin is active).

On top of that base it dispatches beforeEfficacyComposite3d so 3D plugins can draw before the embedding is extracted. Plugins receive: { canvas, ctx, landmarks3d: state.lastLandmarks3d }

The base is copied into a fresh canvas (never mutated in place), so passing the 2D path's canvas here is safe even though 3D plugins may draw on top.

Parameters

• baseCanvas (HTMLCanvasElement | null, optional, default: null) — Pre-composited 2D frame to embed.

Returns

• Promise.<({canvas: HTMLCanvasElement, embedding: Array.<number>}|null)> — Returns null if model not ready.

findFace3d( baseCanvas?: HTMLCanvasElement | null, ): Promise.<({liveInfo3d: object, composite3d: (object|null)}|null)>

Run the full 3D find pipeline:

1. Extract embedding from the current video frame.
2. Find best cosine-similarity match in state.db3d.
3. If a plugin is active, also extract composite embedding.
4. Return raw metrics for the orchestrator to compose into the unified matchStateChanged payload — does NOT dispatch any event.

Returns null if:

• ImageEmbedder not loaded yet
• state.db3d is empty (3D DB has no entries)

Returns { liveInfo3d, composite3d } otherwise, where composite3d is null when no plugin is active.

Parameters

• baseCanvas (HTMLCanvasElement | null, optional, default: null) — 2D-composited frame (video + active 2D Ghostyle) to embed for the composite comparison. Passed straight through to compositeAndDetect3d; when omitted the composite falls back to the raw video frame.

Returns

• Promise.<({liveInfo3d: object, composite3d: (object|null)}|null)>

decideMatchState3d(params: object): Object

Decide match state for the 3D (cosine-similarity) engine.

Cosine similarity convention: HIGHER = MORE SIMILAR. threshold: similarity >= MATCH_THRESHOLD_3D → matched.

When a ghostyle is present, the obfuscated embedding is the primary signal (same rationale as engine.js's decideMatchState for the 2D case).

Parameters

• params (object)
  • params.liveMaxSim (number | null) — Best live similarity (highest in DB).
  • params.liveMaxId (number | null) — ID of best live match.
  • params.obfMaxSim (number | null) — Best composite similarity (null if no plugin).
  • params.obfMaxId (number | null) — ID of best composite match.

Returns

• Object

evaluateMatch3d(result3d: Object): Object | null

Build the mediapipe section of the unified matchStateChanged payload.

NOTE: the field is named mediapipe in the payload (matching the landmark source) because both tasks are powered by MediaPipe. The two models run in the same MediaPipe-initiated pipeline.

Parameters

• result3d (Object) — As returned by findFace3d().

Returns

• Object | null — Returns null if liveInfo3d is fully empty (model not ready / no records).