CounterVid

CounterVid is the official implementation of Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models (EMNLP 2026). It provides counterfactual dataset generation, MixDPO, the Base/SFT/T-pref baselines, inference, and evaluation for Qwen2.5-VL-3B/7B and InternVL3-9B.

What is included

• dataset generation with SigLIP, Claude, Qwen-Image-Edit and Wan2.2;
• generation prompts, ComfyUI API workflows, safety screening and resumable stages;
• canonical visual and textual preference schemas;
• native MixDPO, SFT, T-pref, and Base evaluation profiles;
• independent Qwen and InternVL model adapters;
• Hugging Face dataset loading and validation with optional revisions;
• inference and six-task CounterVid scoring as separate operations;
• explicit adapter/non-LoRA training checkpoints and local merged export.

Model weights, datasets, third-party repositories, historical HPC scripts, and experimental outputs are not distributed in this repository.

Install

Use isolated environments because the Qwen and InternVL runtimes require incompatible Transformers and PEFT versions:

python -m pip install -r requirements/qwen.txt
python -m pip install --no-build-isolation -r requirements/flash-attn.txt
python -m pip install --no-deps -e .

For InternVL, use Python 3.9 and requirements/internvl.txt. A CPU-only environment for validation, tests, and scoring is also provided. See docs/install.md.

Dataset

The public dataset is aimagelab/CounterVid: 26,167 training pairs and 2,910 validation pairs. By default, commands use the current Hugging Face revision. Pass --revision REVISION only when a specific snapshot is needed.

countervid data validate \
  --source aimagelab/CounterVid

The schema and release counts are documented in docs/dataset.md.

Generate a dataset

Install requirements/generation.txt in a separate Python 3.11 environment and configure your source video-caption JSONL, output directory, ComfyUI server and Anthropic API key. Then run:

countervid generate --config configs/generation/countervid.yaml --check
countervid generate --config configs/generation/countervid.yaml --limit 2

The pipeline includes action filtering, bounded edit refinement, Q16/NudeNet screening, preference construction and anchor-disjoint splitting. Its exported dataset can be used directly as dataset.source for training or inference. See docs/generation.md for setup, model filenames, resume behavior and validation.

Train, infer, and score

The supplied configurations use the current Hugging Face model and dataset revisions. For a frozen reproduction, add explicit revision values to the model and dataset sections.

export COUNTERVID_OUTPUT_DIR=/path/to/runs
countervid train --config configs/train/qwen2.5-vl-3b-mixdpo.yaml

export COUNTERVID_MODEL_ID=/path/to/merged-model
countervid infer --config configs/infer/qwen2.5-vl-3b-mixdpo.yaml
countervid evaluate \
  --predictions /path/to/runs/qwen2.5-vl-3b-mixdpo-evaluation/predictions.jsonl \
  --config configs/infer/qwen2.5-vl-3b-mixdpo.yaml

Every training or inference run writes a resolved run_manifest.json; inference writes predictions.jsonl, and scoring writes metrics.json. SimCSE is the explicit offline default. An Anthropic-based factual-consistency scorer is also available and must be selected explicitly; neither scorer is ever used as a silent fallback for the other.

See docs/training.md and docs/evaluation.md.

Checkpoint export

Training outputs are deliberately unambiguous:

• adapter/ contains the LoRA adapter;
• non_lora_trainables.safetensors contains trained projector/merger tensors;
• checkpoint_manifest.json identifies this as an unmerged training checkpoint.

countervid artifacts export creates standalone merged weights locally. It never uploads anything:

countervid artifacts export \
  --config configs/train/qwen2.5-vl-3b-mixdpo.yaml \
  --checkpoint /path/to/run/final \
  --output /path/to/merged-model

Citation

@inproceedings{poppi2026countervid,
  title     = {{CounterVid}: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models},
  author    = {Poppi, Tobia and Uzkent, Burak and Garg, Amanmeet and Porto, Lucas and Kessler, Garin and Yang, Yezhou and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita and Schiffers, Florian},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026}
}

CounterVid is licensed under Apache-2.0. Third-party model and dataset terms remain applicable; see NOTICE.