Modern fisheye dewarping library for the web using WebGPU
npm install @gyeonghokim/fisheye.jsDEMO:
> Modern fisheye undistortion library for the web using WebGPU (general-purpose GPU compute)
fisheye.js processes VideoFrames with WebGPU compute shaders—no canvas 2D—and corrects fisheye lens distortion using the OpenCV fisheye model (Kannala–Brandt–style polynomial in angle θ with coefficients k1–k4). This is the same model as in OpenCV's fisheye module.
- WebGPU GPGPU: Compute-shader pipeline via TypeGPU; input/output as textures and readback to VideoFrame—no canvas element for undistortion
- OpenCV fisheye (Kannala–Brandt) model: Distortion model θ_d = θ × (1 + k1·θ² + k2·θ⁴ + k3·θ⁶ + k4·θ⁸) for accurate calibration
- WebCodecs: Built on the VideoFrame API
- ESM: import { Fisheye } from "@gyeonghokim/fisheye.js"
- npm: Install via npm or other package managers
``bash`
npm install @gyeonghokim/fisheye.jsoptional
npm install --save-dev @webgpu/types
if you installed @webgpu/types,
`json`
{
"compilerOptions": {
"types": ["@webgpu/types"]
}
}
> Why should I install webgpu types?
> This library does not render your binary, it just undistorts the VideoFrame.
> You should make your own YUV renderer, or you can install @gyeonghokim/yuv-player.
in your code,
`ts
import { Fisheye } from "@gyeonghokim/fisheye.js";
// Option 1: Flat style (simple)
const fisheye = new Fisheye({
// OpenCV fisheye distortion coefficients D = [k1, k2, k3, k4]
k1: 0.5,
k2: 0.0,
k3: 0.0,
k4: 0.0,
// Output size
width: 1920,
height: 1080,
// Optional: Camera matrix K parameters
fx: 1000, // focal length x (pixels)
fy: 1000, // focal length y (pixels)
cx: 960, // principal point x (pixels)
cy: 540, // principal point y (pixels)
// Optional: New camera matrix P estimation parameters
balance: 0.0, // 0.0 = no black edges (zoom in), 1.0 = keep original FOV (may have black edges)
fovScale: 1.0, // >1.0 = widen FOV, <1.0 = narrow FOV
// Optional: Projection mode
projection: { kind: "rectilinear" }, // or "equirectangular", "cylindrical", "original"
// Optional: e-PTZ (pan/tilt/zoom) or multi-pane layout (use only one)
// ptz: { pan: 0, tilt: 0, zoom: 1 },
// pane: { kind: "2pane", orientation: "horizontal" }, // or { kind: "4pane" }
});
// Option 2: Grouped style (OpenCV-like)
const fisheyeGrouped = new Fisheye({
K: { fx: 1000, fy: 1000, cx: 960, cy: 540 },
D: { k1: 0.5, k2: 0, k3: 0, k4: 0 },
size: { width: 1920, height: 1080 },
balance: 0.0,
fovScale: 1.0,
projection: { kind: "rectilinear" },
// ptz: { pan: 0, tilt: 0, zoom: 1 } or pane: { kind: "4pane" } (mutually exclusive)
});
// Option 3: Manual rectilinear with explicit P matrix
const fisheyeManual = new Fisheye({
k1: 0.5, k2: 0, k3: 0, k4: 0,
width: 1920,
height: 1080,
projection: { kind: "rectilinear", mode: "manual", newFx: 800, newFy: 800 },
});
const renderLoop = async (timestamp: DOMHighResTimestamp) => {
const undistorted: VideoFrame = await fisheye.undistort(yourVideoFrame);
yourYUVPlayer.draw(undistorted);
requestAnimationFrame(renderLoop);
};
`
This library uses the OpenCV fisheye model (Kannala-Brandt, 2006) for undistortion:
``Normalized coordinates (a, b) where a = X/Z, b = Y/Z
r = sqrt(a² + b²)
θ = atan(r) # incidence angle
θ_d = θ × (1 + k₁θ² + k₂θ⁴ + k₃θ⁶ + k₄θ⁸) # distorted angle
x' = (θ_d / r) × a, y' = (θ_d / r) × b # distorted coords
u = fx(x' + αy') + cx, v = fy × y' + cy # pixel coords
This is the same model as OpenCV's fisheye module.
Important: OpenCV's fisheye.undistortImage() always outputs rectilinear (perspective) projection only. It does not provide panoramic or other projection modes.
For WebGPU efficiency, projection transformation is applied in a single GPU pass:
| Mode | Description |
|------|-------------|
| rectilinear | Standard perspective projection (same as OpenCV) |rectilinear
| + mode: "manual" | Explicit P matrix with newFx, newFy, newCx?, newCy? |equirectangular
| | Panoramic equirectangular projection |cylindrical
| | Panoramic cylindrical projection |original
| | Pass-through (no undistortion) |
Traditional OpenCV approach:
`pythonStep 1: Undistort (GPU/CPU)
undistorted = cv2.fisheye.undistortImage(img, K, D, Knew)
Our approach (single GPU compute shader):
`typescript
// All in one GPU pass: undistortion + projection
const undistorted = await fisheye.undistort(input);
`API
$3
Creates a new Fisheye undistortion instance.
#### OpenCV Fisheye Model Parameters
| Parameter | Type | Default | Description |
| ---------- | --------- | ---------- | -------------------------------------------------------- |
|
fx | number? | auto | Camera matrix K: focal length in x-axis (pixels) |
| fy | number? | auto | Camera matrix K: focal length in y-axis (pixels) |
| cx | number? | width/2 | Camera matrix K: principal point x-coordinate (pixels) |
| cy | number? | height/2 | Camera matrix K: principal point y-coordinate (pixels) |
| k1 | number? | 0 | Distortion coefficient k1 (Kannala-Brandt) |
| k2 | number? | 0 | Distortion coefficient k2 (Kannala-Brandt) |
| k3 | number? | 0 | Distortion coefficient k3 (Kannala-Brandt) |
| k4 | number? | 0 | Distortion coefficient k4 (Kannala-Brandt) |
| width | number? | 300 | Output image width (OpenCV new_size.width) |
| height | number? | 150 | Output image height (OpenCV new_size.height) |
| balance | number? | 0.0 | Balance (0.0 = no black edges/zoom in, 1.0 = keep original FOV) |
| fovScale | number? | 1.0 | FOV scale (>1.0 = widen FOV, <1.0 = narrow FOV) |Note: These parameters exactly match OpenCV fisheye API. Use values from
cv2.fisheye.calibrate() or cv2.fisheye.estimateNewCameraMatrixForUndistortRectify().#### Mode options (e-PTZ vs multi-pane)
You can optionally use e-PTZ (electronic pan/tilt/zoom) for a single view, or multi-pane for split views. Use only one of
ptz or pane; they are mutually exclusive.| Parameter | Type | Description |
| --------- | ---- | ----------- |
|
ptz | PTZOptions? | e-PTZ: pan (degrees, -180–180), tilt (-90–90), zoom (1.0 = no zoom, >1 = zoom in). |
| pane | PaneLayout? | Multi-view: { kind: "2pane", orientation?: "horizontal" \| "vertical" } or { kind: "4pane" }. |- PTZOptions:
{ pan?: number; tilt?: number; zoom?: number }
- PaneLayout: TwoPaneLayout (2-pane split, optional orientation) or FourPaneLayout ({ kind: "4pane" })#### Projection Options
| Parameter | Type | Default | Description |
| ------------ | -------------------- | -------------------------- | -------------------------------------------------------------- |
|
projection | FisheyeProjection | { kind: "rectilinear" } | Output projection mode (see Projection Modes above) |FisheyeProjection types:
`typescript
// Auto: P matrix computed from balance/fovScale
{ kind: "rectilinear" }
{ kind: "equirectangular" }
{ kind: "cylindrical" }
{ kind: "original" }// Manual: Explicit P matrix
{ kind: "rectilinear", mode: "manual", newFx: number, newFy: number, newCx?: number, newCy?: number }
`$3
Undistorts a VideoFrame with fisheye distortion.
Parameters:
-
frame: Input VideoFrame with fisheye distortionReturns: Promise that resolves to an undistorted VideoFrame
Differences from OpenCV:
Unlike OpenCV's
fisheye.undistortImage() which only outputs rectilinear (perspective) projection, this method performs all transformations in a single GPU pass for WebGPU efficiency:1. Undistortion (OpenCV Kannala-Brandt fisheye model)
2. Projection (rectilinear, equirectangular, cylindrical, or original)
OpenCV equivalent for non-rectilinear projections would require 2 separate operations:
`python
OpenCV: 2 CPU/GPU roundtrips for panoramic projection
undistorted = cv2.fisheye.undistortImage(img, K, D, Knew)
panorama = cv2.remap(undistorted, map_x, map_y, cv2.INTER_LINEAR)fisheye.js: 1 GPU pass - undistortion + projection
undistorted = await fisheye.undistort(img)
`$3
Updates the configuration. You can update any subset of the original options.
$3
Cleans up GPU resources. Call this when you're done using the instance.
Working with YUV Binary Data
If you receive raw YUV binary data from a camera or server, you can use the
createVideoFrameFromYUV utility to create a VideoFrame:`ts
import { Fisheye, createVideoFrameFromYUV } from "@gyeonghokim/fisheye.js";const fisheye = new Fisheye({ k1: 0.5, width: 1920, height: 1080 });
// Example: Receiving NV12 data from a server
const response = await fetch("/api/camera/frame");
const yuvBuffer = await response.arrayBuffer();
const frame = createVideoFrameFromYUV(new Uint8Array(yuvBuffer), {
format: "NV12", // YUV format
width: 1920,
height: 1080,
timestamp: performance.now() * 1000, // microseconds
});
const undistorted = await fisheye.undistort(frame);
frame.close(); // Don't forget to close the original frame
`$3
Creates a VideoFrame from YUV binary data.
Parameters:
-
data: YUV binary data (ArrayBuffer, TypedArray, or DataView)
- options: Configuration object
- format (required): YUV pixel format
- width (required): Frame width in pixels
- height (required): Frame height in pixels
- timestamp (required): Timestamp in microseconds
- duration (optional): Duration in microseconds
- displayWidth (optional): Display width (defaults to width)
- displayHeight (optional): Display height (defaults to height)
- colorSpace (optional): Color space configuration
- transfer (optional): If true, transfers buffer ownership for zero-copy performanceSupported YUV Formats:
| Format | Description | Data Size |
| ------- | ----------------------------------------------- | -------------------- |
|
I420 | YUV 4:2:0 planar (Y, U, V planes) | width × height × 1.5 |
| NV12 | YUV 4:2:0 semi-planar (Y plane, interleaved UV) | width × height × 1.5 |
| I420A | YUV 4:2:0 planar with alpha | width × height × 2.5 |
| I422 | YUV 4:2:2 planar | width × height × 2 |
| I444 | YUV 4:4:4 planar | width × height × 3 |$3
Calculates the expected byte size for YUV data.
`ts
import { calculateYUVDataSize } from "@gyeonghokim/fisheye.js";const size = calculateYUVDataSize("NV12", 1920, 1080); // 3110400 bytes
`Development
This project uses:
- Biome for linting and formatting
- Husky for git hooks
- Commitlint for conventional commit messages
- Semantic Release for automated versioning and publishing
- TypeScript for type safety
- WebGPU for GPU-accelerated processing
$3
`bash
npm run build # Build the library
npm run dev # Build in watch mode
npm run lint # Run linter
npm run lint:fix # Fix linting issues
npm run format # Format code
npm run type-check # Check TypeScript types
`$3
This project follows the Conventional Commits specification:
`
: [optional body]
[optional footer]
`Types:
feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert`MIT