Vladyslav Kovalchuk
← All cases

Engineering an ASCII video background: from MP4 to canvas at 79 KB gzipped

Live ↗

Problem

I wanted ambient motion in the background of my portfolio home page — something memorable but not heavy. The obvious option was a looping <video> tag, but that ships a multi-megabyte payload, decodes through the browser's media stack with its own paint and compositor cost, refuses to theme with the rest of the page, and aesthetically clashes with the deliberately minimal monospace tone of a developer portfolio. I wanted an ASCII rendition of a generated clip — visually distinct, type-coherent with the surrounding copy, and deliberately small as a file artifact. The challenge: encode 240 frames of motion as text data, paint them into a canvas without dropping frames or burning CPU, and integrate the result as background art that respects theme, scroll, idle state, viewport size, and Retina pixel density.

Constraints

  • Bandwidth budget on a portfolio is tight — viewers may bounce in seconds, so every KB on the critical path matters.
  • Idle CPU usage must be near zero. This is decorative; it cannot fight the page for resources or burn mobile battery.
  • Must look sharp on both Retina and standard displays without shipping per-DPR assets.
  • Must track the site's light/dark toggle instantly, with no flash and no remount.
  • The source clip carried a Gemini watermark sparkle in the bottom-right that had to disappear without leaving a visible hole in the lily silhouette behind it.
  • Cannot block hydration — the animation must not delay the first paint of the actual hero copy.

Solution

Two halves. First, a Node script that uses fluent-ffmpeg to pipe the source MP4 through a drawbox filter (to mask the watermark in source pixel space), then a Lanczos downscale to a 120x40 cell grid, then format=gray to emit raw 8-bit luma samples. The script reads those bytes directly — no PNG round-trip — maps them through a 10-step brightness ramp (' .:!ilMW@U+2588'), and packs the output into a JSON blob with runs of three or more spaces inline-RLE-encoded as ~N|. Result: 410 KB raw, 79 KB gzipped on the wire. Second, a client React component fetches that JSON via requestIdleCallback, decodes each row's RLE once into memory, and paints the animation into a transparent canvas. Each glyph is drawn centered in its grid cell via textAlign: center + textBaseline: middle, so the row lands flush against the canvas edges (the natural fillText alternative leaves a ~4% inset gap on the right, which was an early bug that cropped both the watermark area and the lily's right-most tendrils). The animation uses requestAnimationFrame with a delta-time check that throttles to the source fps. A MutationObserver on <html> re-reads currentColor on theme toggle and repaints the active frame in place. Visibilitychange pauses the loop without time drift. On unmount, RAF is cancelled, listeners are removed, and the decoded frames array is zeroed out so V8 can GC the strings.

Key technical decisions

Canvas + pre-encoded ASCII over an HTML5 <video> tag
The same generated clip as an mp4 was ~1.8 MB even after a tight HandBrake pass — over 20 times the wire weight of the ASCII payload, plus a media-decode loop that is not free on lower-end devices. A <video> element cannot theme its content, cannot blend semantically with surrounding monospace type, and its compositor layer permanently promotes a portion of the page out of CPU paint into the GPU compositor, which is great for full-screen playback but wasteful for a 600x340 corner widget. The trade-off is fidelity — ASCII loses color and pixel detail — but for a deliberately stylised piece that is actually a feature. Rule of thumb I now use: full-bleed real-world footage or anything users will pause and seek stays as <video>; short-loop stylistic ambient art with a flat color palette becomes canvas ASCII.
fluent-ffmpeg piped to stdout, no PNG round-trip
Most ASCII video tutorials extract frames to disk as PNG then re-decode each one to read pixels. That is 240 file writes plus 240 PNG decompressions for no reason. Asking ffmpeg for -f rawvideo -pix_fmt gray gives back exactly what is needed — one 8-bit luma byte per pixel — over a single readable stream. The whole 240-frame extraction takes ~220 ms on this hardware, and the encode/RLE pass takes ~12 ms. For alternative tooling I evaluated: jp2a and ascii-image-converter are excellent for stills but cannot stream video at the resolution I needed; ImageMagick + a custom mapper works but is dramatically slower per frame; pure-JS PNG decoders (pngjs) add a real dependency and round-trip I do not need. fluent-ffmpeg here is just a thin spawn wrapper, and that is the right level of abstraction.
Brightness ramp design: sparse middle, dense peaks, monospace-friendly glyphs
Classic Bradford ramps run 70 characters from black to white. At a 10 px-per-cell downscale that resolution is wasted — too many glyph candidates with too little luma difference, which reads as noise rather than gradient. I picked a 10-step ramp: a true space for black, then .:!il for edges and medium tones (vertical-emphasis glyphs that visually read as ink strokes following the petal lines), then MW@ for solid bright fills, then U+2588 FULL BLOCK for the peaks. Vertical-emphasis glyphs on the edges follow the lily's bloom lines; the full block punctuates the brightest pixels without a hash sign, which always reads as code, not as art. ~ and | are intentionally absent from the ramp so they can be used as inline RLE markers without ambiguity.
Per-cell RLE only on spaces, trust gzip for the rest
Most of the frame is black, which maps to spaces. Replacing runs of three or more spaces with the marker ~N| shaves ~40% off the raw JSON before any HTTP compression — and importantly it pays off most exactly where gzip already struggles (long uniform runs are well-handled, but skipping them entirely is cheaper on the decode side). Beyond that, I did not write a binary encoder. Gzip and Brotli already handle character-level redundancy well, an encoded binary format would require shipping a decoder, and the JSON pipeline lets browsers stream and JSON.parse it natively. 410 KB raw, 79 KB gzip on the wire, ~1 MB decoded into strings client-side — that is a 5x compression at zero client cost.
Per-glyph fillText, not per-row
The natural way to draw an ASCII row is ctx.fillText(rowString, 0, y) — fast, one call per row. The catch is that monospace advance is roughly 0.6 of font size, so the rightmost glyph of a 120-character row lands around 96% of canvas width, not at the edge. Invisible on a centered subject; very visible on a clip where the lily branches and the Gemini watermark hugged the right edge. Switching to textAlign: center + textBaseline: middle and drawing each glyph at the precise center of its grid cell costs ~120x40 = 4,800 fillText calls per frame x 24 fps = ~115k calls/sec, which modern Canvas 2D handles in 3-4 ms per frame with alpha: true. The right-edge alignment is now pixel-perfect. The lesson generalises: font advance metrics will not align to whatever grid you want — if you need a grid, draw a grid.
Mask the watermark in source pixel space, not after the ASCII encode
The clip carried a Gemini sparkle in the bottom-right corner. Erasing it after the encode would leave a visible hole in every frame where the lily bloom reached that area. Adding drawbox=x=1130:y=545:w=60:h=110:color=black:t=fill as the first ffmpeg filter paints a solid black rectangle over the watermark in the original 1280x720 source, which then blends invisibly into the black background after Lanczos downscale. No artifact, no lily content lost, and the mask coordinates live in source pixel space where they are easy to reason about and to verify by inspecting frames pre- and post-mask.
Z-index inversion + backdrop-blur per text block, not per section
First version had the canvas at z-50 and a translucent backdrop-blur card wrapping each section. That backwards: the canvas painted over the text instead of behind it, and the section-wide cards put blur in the empty space between text blocks instead of just behind the text itself. Fix: canvas at z-0, <main> and <footer> claim z-10 so positioned text always wins the stacking battle, and the blur halo is moved to individual text elements as inline-block w-fit rounded-md bg-background/60 backdrop-blur-[2px]. Each h1 line, paragraph, link, and tech tag gets a tight halo that hugs the glyphs; the gaps between blocks let the ASCII bloom through cleanly. Same trick keeps text legibility intact on the busy art without painting a full sheet of glass over the page.
Defer fetch to idle, pause on hidden, drop refs on unmount
Loading and decoding a megabyte of strings on hydration is exactly the kind of work that delays first paint of the actual hero copy. The component defers fetch via requestIdleCallback with a 1500 ms timeout, with setTimeout(200) as a Safari fallback. A visibilitychange listener pauses the RAF when the tab is hidden and resets lastFrameTime on return so the animation does not fast-forward through skipped time. On unmount, RAF is cancelled, both listeners are removed, the MutationObserver is disconnected, and the frames array is reassigned to [] so the ~1 MB of decoded string data can be GC'd. None of these by themselves are heroic — they are table stakes for code that lives in a page someone has open in a tab they forgot about.
DPR capped at 2, glyph color read from CSS
Retina screens report devicePixelRatio of 2 or 3. Rendering at 3x means drawing 9 times the pixels with no perceptual gain on text-sized glyphs — capping at 2 is the standard trick. For theme, instead of wiring a React context into the render path, the canvas wrapper just inherits a Tailwind text-zinc-700 / dark:text-zinc-200 class and the renderer reads getComputedStyle(wrap).color once on init and on theme change. This keeps the canvas decoupled from the React theme system and means future palette tweaks happen in markup, not in code.

Outcome

410 KB JSON / 79 KB gzipped on the wire, decoded once into ~1 MB of strings client-side and never re-allocated. Paint cost is ~3-4 ms per frame at 24 fps on a 1280x720 effective canvas, with the loop yielding ~98% idle between renders. Lighthouse scores did not move — no LCP delay (the canvas waits for idle), no CLS (it is aria-hidden, fixed-position, outside flow). Same payload and same renderer drive both the corner widget on / and the full-screen preview at /ascii. The biggest lesson was the right-edge fillText cropping: a reminder that drawing primitives in Canvas 2D are sharper instruments than they look, and that the right time to verify pixel-level alignment is before you ship — not after the user with a generated source clip points at the missing sparkle in the bottom-right corner.