Vladyslav Kovalchuk
← All cases

Engineering an ASCII video background: from MP4 to canvas at 79 KB gzipped

Running on the home page ↗

Problem

I wanted ambient motion in the background of my portfolio home page — something memorable but not heavy. The obvious option was a looping <video> tag, but that ships a multi-megabyte payload, decodes through the browser's media stack with its own paint and compositor cost, refuses to theme with the rest of the page, and aesthetically clashes with the deliberately minimal monospace tone of a developer portfolio. I wanted an ASCII rendition of a generated clip — visually distinct, type-coherent with the surrounding copy, and deliberately small as a file artifact. The challenge: encode 240 frames of motion as text data, paint them into a canvas without dropping frames or burning CPU, and integrate the result as background art that respects theme, scroll, idle state, viewport size, and Retina pixel density.

Constraints

  • Bandwidth budget on a portfolio is tight — viewers may bounce in seconds, so every KB on the critical path matters.
  • Idle CPU usage must be near zero. This is decorative; it cannot fight the page for resources or burn mobile battery.
  • Must look sharp on both Retina and standard displays without shipping per-DPR assets.
  • Must track the site's light/dark toggle instantly, with no flash and no remount.
  • The source clip carried a Gemini watermark sparkle in the bottom-right that had to disappear without leaving a visible hole in the lily silhouette behind it.
  • Cannot block hydration — the animation must not delay the first paint of the actual hero copy.

Solution

Two halves. First, a Node script that uses fluent-ffmpeg to pipe the source MP4 through a drawbox filter (to mask the watermark in source pixel space), then a Lanczos downscale to a 120x40 cell grid, then format=gray to emit raw 8-bit luma samples. The script reads those bytes directly — no PNG round-trip — maps them through a 10-step brightness ramp (' .:!ilMW@U+2588'), and packs the output into a JSON blob with runs of three or more spaces inline-RLE-encoded as ~N|. Result: 410 KB raw, 79 KB gzipped on the wire. Second, a client React component fetches that JSON via requestIdleCallback, decodes each row's RLE once into memory, and paints the animation into a transparent canvas. Each glyph is drawn centered in its grid cell via textAlign: center + textBaseline: middle, so the row lands flush against the canvas edges (the natural fillText alternative leaves a ~4% inset gap on the right, which was an early bug that cropped both the watermark area and the lily's right-most tendrils). The animation uses requestAnimationFrame with a delta-time check that throttles to the source fps. A MutationObserver on <html> re-reads currentColor on theme toggle and repaints the active frame in place. Visibilitychange pauses the loop without time drift. On unmount, RAF is cancelled, listeners are removed, and the decoded frames array is zeroed out so V8 can GC the strings.

Key technical decisions

Canvas + pre-encoded ASCII over an HTML5 <video> tag
The same generated clip as an mp4 was ~1.8 MB even after a tight HandBrake pass — over 20 times the wire weight of the ASCII payload, plus a media-decode loop that is not free on lower-end devices. A <video> element cannot theme its content, cannot blend semantically with surrounding monospace type, and its compositor layer permanently promotes a portion of the page out of CPU paint into the GPU compositor, which is great for full-screen playback but wasteful for a 600x340 corner widget. The trade-off is fidelity — ASCII loses color and pixel detail — but for a deliberately stylised piece that is actually a feature. Rule of thumb I now use: full-bleed real-world footage or anything users will pause and seek stays as <video>; short-loop stylistic ambient art with a flat color palette becomes canvas ASCII.
fluent-ffmpeg piped to stdout, no PNG round-trip
Most ASCII video tutorials extract frames to disk as PNG then re-decode each one to read pixels. That is 240 file writes plus 240 PNG decompressions for no reason. Asking ffmpeg for -f rawvideo -pix_fmt gray gives back exactly what is needed — one 8-bit luma byte per pixel — over a single readable stream. The whole 240-frame extraction takes ~220 ms on this hardware, and the encode/RLE pass takes ~12 ms. For alternative tooling I evaluated: jp2a and ascii-image-converter are excellent for stills but cannot stream video at the resolution I needed; ImageMagick + a custom mapper works but is dramatically slower per frame; pure-JS PNG decoders (pngjs) add a real dependency and round-trip I do not need. fluent-ffmpeg here is just a thin spawn wrapper, and that is the right level of abstraction.
Per-glyph fillText, not per-row
The natural way to draw an ASCII row is ctx.fillText(rowString, 0, y) — fast, one call per row. The catch is that monospace advance is roughly 0.6 of font size, so the rightmost glyph of a 120-character row lands around 96% of canvas width, not at the edge. Invisible on a centered subject; very visible on a clip where the lily branches and the Gemini watermark hugged the right edge. Switching to textAlign: center + textBaseline: middle and drawing each glyph at the precise center of its grid cell costs ~120x40 = 4,800 fillText calls per frame x 24 fps = ~115k calls/sec, which modern Canvas 2D handles in 3-4 ms per frame with alpha: true. The right-edge alignment is now pixel-perfect. The lesson generalises: font advance metrics will not align to whatever grid you want — if you need a grid, draw a grid.
Defer fetch to idle, pause on hidden, drop refs on unmount
Loading and decoding a megabyte of strings on hydration is exactly the kind of work that delays first paint of the actual hero copy. The component defers fetch via requestIdleCallback with a 1500 ms timeout, with setTimeout(200) as a Safari fallback. A visibilitychange listener pauses the RAF when the tab is hidden and resets lastFrameTime on return so the animation does not fast-forward through skipped time. On unmount, RAF is cancelled, both listeners are removed, the MutationObserver is disconnected, and the frames array is reassigned to [] so the ~1 MB of decoded string data can be GC'd. None of these by themselves are heroic — they are table stakes for code that lives in a page someone has open in a tab they forgot about.

Outcome

410 KB JSON / 79 KB gzipped on the wire, decoded once into ~1 MB of strings client-side and never re-allocated. Paint cost is ~3-4 ms per frame at 24 fps on a 1280x720 effective canvas, with the loop yielding ~98% idle between renders. Lighthouse scores did not move — no LCP delay (the canvas waits for idle), no CLS (it is aria-hidden, fixed-position, outside flow). It is running behind this page right now. The biggest lesson was the right-edge fillText cropping: a reminder that drawing primitives in Canvas 2D are sharper instruments than they look, and that the right time to verify pixel-level alignment is before you ship — not after the user with a generated source clip points at the missing sparkle in the bottom-right corner.