        BMVC 2026
        Video Tokenisation
      

Adaptive Tokenisation via Temporal Redundancy Masking and Latent Inpainting

A video tokeniser that decides its own token budget from the encoder pass alone — no router, no scorer, no extra decoder pass. 31× faster allocation than search-based adaptive tokenisers at a matched keep rate.

Kevin Dave^*1 Sai Aditya Patkuri^*1,2 Chhaya Kumar Das^*1 Gouranga Bala¹ Rajesh Kumar S A¹ R. Venkatesh Babu²

¹ Phronetic AI ² Vision and AI Lab (VAL), Indian Institute of Science, Bangalore ^* Equal contribution

arXiv Paper PDF Poster Code Coming soon {} BibTeX

01  /  The budget follows the content

Token usage over time for one clip: static intervals collapse towards zero, motion and scene cuts spike. — One clip, one fixed threshold. The retained fraction tracks the content: static intervals collapse towards zero, motion and scene cuts spike to full rate. Per-video drop rates span **5.15% to 86.10%** on UCF-101 — without any per-clip search.

02  /  Abstract

Video tokenisers spend a fixed token budget on every clip: a static shot and a fight scene cost exactly the same. Adaptive tokenisers fix that, but pay a search or routing tax — ElasticTok runs a binary search over log₂N decoder passes, InfoTok needs an extra full-rate decoder pass.

We show that overhead is unnecessary. The latent space of a frozen continuous tokeniser already exposes temporal redundancy: if a latent position barely changed since the last kept frame, drop it — and inpaint it back before decoding. A parameter-free temporal-L1 criterion produces the mask directly from the encoder output, and a 2.7 M-parameter Latent Inpainting Transformer (LIT) refills the dropped positions in the backbone's native latent space.

The result is a tokeniser whose compression rate emerges from the video rather than being imposed on it, controlled by a single interpretable scalar τ, with zero extra forward evaluations to allocate the budget.

Parameter free

No learned router or scorer. One interpretable scalar, calibrated once per backbone.

Single pass

The mask needs only z — zero extra forward evaluations to allocate the budget.

Content adaptive

The compression rate emerges from the video rather than being imposed on it.

03  /  Method

Temporal-L1 masking

Each latent position is judged against its own history: how far has it moved since the last time this position was kept? The distance is a channel-averaged absolute difference — nothing learned, nothing tuned.

$$\Delta(t',y,x) \;=\; \frac{1}{C}\sum_{c=1}^{C}\bigl| z[c,t',y,x] - z[c,\rho(y,x,t'),y,x] \bigr|$$

$$m(t',y,x) \;=\; \mathbb{1}\bigl[\, \Delta(t',y,x) \ge \tau \,\bigr]$$

where $\rho(y,x,t')$ is the most recently kept temporal position at $(y,x)$, and $C$ is the latent channel count.

keep the position moved by at least τ

drop it moved less than τ — the last kept value still stands in for it

That is the whole criterion. τ is a single scalar, calibrated once per backbone — 0.3 for Cosmos, 1.2 for Omni — and the full formulation is in the paper.

Latent Inpainting Transformer

LIT refills the dropped positions before decoding, using interleaved spatial and temporal attention with 2D RoPE instead of full 3-D attention.

≈9×

cheaper than full 3-D attention

2.7 M

trainable parameters

Because it is supervised on the latents as well as the decoded pixels, its output stays in the backbone's native latent space — a drop-in for anything that consumes those latents.

Algorithm 1 Latent Inpainting Transformer (LIT) pipeline

Require: frozen encoder $\mathcal{E}$; frozen decoder $\mathcal{D}$; threshold $\tau$; LIT parameters $\theta$

Inference is a single encoder pass, a threshold comparison, and one LIT forward — the mask costs nothing beyond the encoding that was already happening.

04  /  Allocation cost

Method
NFE ↓
Secs ↓
Speed-up ↑

{{ r.nfe }}

{{ r.secs }}

{{ r.speedup }}

Per clip on a single A10G. NFE counts the extra forward evaluations needed to set the budget. Ours needs none — the mask falls out of the encoder pass.

05  /  One knob: τ

Raising τ peels the static background away first and keeps the moving subject longest. Drag to sweep the threshold on a real clip.

τ = {{ tauLabel }}

            {{ tauMin }}
            {{ tauMax }}
          

Backbone

Each backbone has its own τ scale — Omni's latents have ≈2.4× the per-channel spread, so the same masking score sits higher.

Mask at the selected threshold, time running left to right — kept dropped {{ backboneLabel }} backbone. The left-most panel is the input frame; the rest are the mask at successive latent timesteps, with the retained fraction printed beside τ.

06  /  Rate–distortion

Sweeping the single threshold traces a smooth, monotonic rate–distortion curve on DAVIS — no retraining, one calibration per backbone. Red annotations give the change from the neighbouring operating point.

PSNR, SSIM, LPIPS and FVD against keep rate for a sweep over the threshold — {{ rdCaption }}

07  /  Reconstructions

At 21–61% of the tokens we stay on par with ElasticTok at 80–100%, and with the full-rate Cosmos backbone. Drag the handle to wipe between the ground truth and the selected method.

↔

Ground truth

{{ methodLabel }}

Three UCF-101 clips, three frames each. Our keep rates are printed on the first frame of each clip.

08  /  Reconstruction quality

Click a column header to sort

{{ r.keep }}

{{ r.psnr }}

{{ r.ssim }}

{{ r.lpips }}

{{ r.fvd }}

Baselines are matched to us — InfoTok to our emergent keep rate, ElasticTok-CV to a strict error threshold. At an identical budget we gain +2.81 dB (TokenBench) and +2.47 dB (DAVIS) over InfoTok.

09  /  Streaming & ablations

Token usage against time without caching: regular spikes to 100% — Same 27-second video, same τ. Without the cache, token usage snaps back to 100% on a regular beat even though the content has barely changed; with it, the rest of the video streams at roughly 20% of the tokens.

Token usage against time with a cached reference: flat at roughly 20% — Same 27-second video, same τ. Without the cache, token usage snaps back to 100% on a regular beat even though the content has barely changed; with it, the rest of the video streams at roughly 20% of the tokens.

Ablations — DAVIS, 62% keep rate

Variant
PSNR↑
SSIM↑
LPIPS↓
FVD↓

{{ r.psnr }}

{{ r.ssim }}

{{ r.lpips }}

{{ r.fvd }}

Fidelity comes from the inpainting: LIT gains +0.7 dB and −170 FVD over copying the last kept latent.

10  /  Masks in the wild

Kept and dropped tokens over time for the selected clip — The first panel is the input frame; the rest are the mask at successive latent timesteps at the operating threshold. Structure the tokeniser never needs to re-send — court markings, walls, foliage — drops out immediately, while the moving subject stays.

11  /  BibTeX

@article{dave2026adaptive,
  title     = {Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting},
  author    = {Dave, Kevin and Patkuri, Sai Aditya and Das, Chhaya Kumar and Bala, Gouranga and Babu, R Venkatesh and SA, Rajeshkumar},
  journal   = {arXiv preprint arXiv:2606.06158},
  year      = {2026}
}

Temporal-L1 masking

Latent Inpainting Transformer

Streaming with a cached reference

Ablations — DAVIS, 62% keep rate