AMD Hackathon · Track 2 · Video Captioning

StyleFrame turns video clips into captions with four distinct voices.

A linux/amd64 Docker agent that samples video frames, compresses temporal context into a contact sheet, asks a Fireworks multimodal model for grounded visual facts, and writes the required caption JSON.

Copy Docker image See pipeline
t+01
t+18
t+36
t+54

CONTACT SHEET

→

MINIMAX-M3

→

CAPTIONS

4

caption styles per clip

49.5s

local benchmark on 3 clips

4.35/5

local LLM-style judge score

amd64

container-ready image

Architecture

The agent treats a video as a compact visual timeline.

tasks.jsonread input
FFmpegsample frames
Contact Sheetcompress context
Fireworksvision reasoning
Guardrailsstyle + JSON checks
results.jsonwrite output

Live demo

Paste a video URL and generate captions online.

The browser samples frames into a contact sheet, then Vercel securely calls Fireworks from a server-side API route. Your API key never enters the browser.

Works best with CORS-enabled MP4 URLs. The official public sample above is ready to try.

Ready. Paste a video URL and press Generate.

Sample clip

One scene, four tones.

The UI below is a front-end demo of the public sample output. Hidden evaluation still runs the Docker image directly with mounted input and output folders.

formal task_id: v1

A busy urban street with two-way traffic and autumn trees under daytime lighting.

urban street autumn foliage moving traffic daylight

Quality controls

Small guardrails make the captions more judge-friendly.

Grounding first

Frames become visual facts before style generation, reducing captions that drift away from the scene.

Style separation

Tech humor must contain technical language; non-tech humor is kept plain and everyday.

Runtime aware

Contact sheets keep multimodal calls compact enough for the 10-minute hidden evaluation limit.

Schema safe

Output is normalized into the required JSON shape so missing styles do not silently break scoring.

Submission image

Run the same container used for evaluation.

ghcr.io/noah-wang/amd-video-caption-agent:latest
docker run --rm --platform linux/amd64 \
  -e FIREWORKS_API_KEY="$FIREWORKS_API_KEY" \
  -v "$(pwd)/input:/input" \
  -v "$(pwd)/output:/output" \
  ghcr.io/noah-wang/amd-video-caption-agent:latest