Anticipate: Predictive Urban Safety with Foundation-Model Video Intelligence

Context

Built in 24 hours at NVIDIA Spark Hack NYC 2026 on an NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory). Anticipate detects dangerous physical dynamics in surveillance footage and generates structured incident reports, entirely on-device, so no video frame ever leaves the machine. I owned the ML pipeline and backend engineering on a 4-person team.

Problem

Urban surveillance produces far more footage than humans can watch, and privacy regulations often prohibit camera footage from leaving the premises. How do you build a video anomaly detection system that is fast enough for live streams, generalizes to scenes it was never trained on, and runs fully on-device with zero cloud dependency?

Approach

Used Meta's V-JEPA 2 self-supervised video world model (326M params, BF16) as a frozen encoder and designed a custom attentive probe head on top (LayerNorm, a learnable query token, multi-head cross-attention, and a feed-forward scorer; 4.86M params), betting that attentive pooling over the 8,192 V-JEPA tokens would generalize across domains instead of memorizing training artifacts. Trained the probe on UCF-Crime embeddings in ~3 minutes on the GB10, then wired the pipeline into a FastAPI + WebSocket server with live HLS ingestion from NYC DOT traffic cameras and an on-device LLM report generator.

Frameworks

Self-Supervised Video Foundation Models (V-JEPA 2)Frozen Encoder + Trainable ProbeAttentive Pooling / Cross-AttentionBF16 Mixed-Precision TrainingReal-Time Stream Processing

Implementation

  • •V-JEPA 2 encoder wrapper validated at 347 ms per 64-frame clip on the GB10
  • •Custom attentive probe head (4,857,857 params) with FP32 BCELoss cast for BF16 numerical stability
  • •Cached feature-extraction pipelines for mp4 and PNG-frame datasets (~1.1 s and ~410 ms per clip)
  • •FastAPI + WebSocket server streaming telemetry and risk scores to a Next.js dashboard
  • •Live HLS ingestion from three real NYC DOT traffic cameras, processed continuously in 5-second chunks
  • •On-device Llama-3.1-8B report generator producing schema-constrained JSON incident reports
  • •Docker + docker-compose deployment on the NVIDIA NGC PyTorch container with GPU passthrough

Outcomes

  • ✓End-to-end clip-to-risk-score latency of ~1.1 seconds, dominated by video decode rather than GPU inference
  • ✓Cross-domain generalization demo: probe trained only on UCF-Crime fired correctly on 8 of 15 unseen clips across subway platforms, intersections, and construction sites
  • ✓Out-of-distribution sanity check: synthetic test video scored 0.0025, suggesting the probe learned generalizable anomaly features
  • ✓Live monitoring worked end-to-end on three real traffic cameras with varying risk scores across traffic conditions
  • ✓Full privacy architecture: only structured JSON reports leave the device, never video

Learnings

  • →Frozen foundation model + small trainable head is a remarkably fast path to cross-domain video understanding
  • →End-to-end latency budgets are won or lost in decode and I/O, not model inference
  • →BF16 training needs deliberate numerical-stability workarounds (FP32 casts in loss functions)
  • →A 24-hour hackathon demo is evidence of feasibility, not production readiness; the honest framing matters