Group-validity classification

Code for the group-validity classification task: deciding whether a group of product listings contains a duplicate (two listings of the same product, ignoring size).

Method

The proposed method runs end-to-end across four notebooks at the repository root:

1. data_preparation.ipynb — split the items by product label, normalize price per geo, one-hot/multi-hot encode geo, color and department tags, embed titles and descriptions with LaBSE and images with fashion-CLIP, and save the preprocessed splits.
2. training.ipynb — train the contrastive attention encoder that fuses title/description/image embeddings (SupCon loss), then mine item pairs and train the MLP classifier that scores a pair as same-product, picking the best decision threshold on the validation set.
3. attention.ipynb — load the trained encoder and export its per-modality attention weights (title/description/image) for analysis.
4. evaluation.ipynb — retrain encoder + classifier over 10 seeds and evaluate on every test split (sizes 5–10, hard, category), reporting F1/accuracy mean ± std and the concatenated-over-runs F1.

Baselines

Baseline methods and evaluation scripts for the group-validity classification task. All scripts are run from the repository root and write predictions to results/<timestamp>/.

Expected data

The scripts read from a data/ directory at the repository root (not included in this repository yet):

• train_split.csv, val_split.csv, test_split.csv — item-level metadata splits
• embeddings_train.npy, embeddings_val.npy, embeddings_test.npy — CLIP image + textual embeddings
• item_ids_train.npy, item_ids_val.npy, item_ids_test.npy — item IDs aligned with the embedding rows
• labels_train.npy, labels_val.npy — per-item group IDs aligned with the embedding rows (used to train the CLIP + contrastive projection baseline)
• train_groups_{5..10}.csv — train group tasks (items sampled from the train+val pool)
• groups_{5..10}.csv — test group tasks of size 5–10
• groups_hard_5.csv, groups_category_5.csv — hard and same-category test variants

Baselines

Random

Assigns a uniform random score to each group, sweeps the decision threshold for best train F1, and evaluates on test over 10 random seeds:

python baselines/random_baseline.py

CLIP-only

Scores each group by the maximum pairwise cosine similarity of its CLIP embeddings, sweeps 500 threshold candidates (evenly spaced between the min and max train score) for best train F1, and evaluates on test with 10 bootstrap repeats:

python baselines/clip_only_baseline.py

CLIP + contrastive projection

A small MLP (input_dim → hidden_dim → LayerNorm → ReLU → Dropout → output_dim) projects the raw CLIP embeddings into a lower-dimensional latent space, trained with a supervised contrastive (InfoNCE) loss so that items in the same product group are pulled together. We tested the following hyperparameters:

• hidden_dim=[712, *512*, 256, no_hidden_layer]
• output_dim=[*192*, 256]
• batch_size=[*16384*, 8192, 4096]
• lr=[*1e-4*, 1e-5]
• dropout=[0.0, *0.2*, 0.3, 0.5]
• n_thresholds=100
• layer_norm=[true, false]

Groups are then scored exactly like the CLIP-only baseline — maximum pairwise cosine similarity of the projected embeddings — with a threshold swept on train (100 candidates by default, --n-thresholds) and evaluated on test over 10 bootstrap repeats. Lives in baselines/clip_contrastive/.

1. Train the projection model on item-level embeddings + per-item group labels (the group ID each item belongs to), saving model.pt + config.json to --out-dir. Defaults reproduce the reported best configuration:

   python baselines/clip_contrastive/train.py \
       --embeddings data/embeddings_train.npy --labels data/labels_train.npy \
       --val-embeddings data/embeddings_val.npy --val-labels data/labels_val.npy \
       --out-dir baselines/models/clip_contrastive

2. Evaluate: load the trained model from --model-dir, project the embeddings, and score all test splits (sizes 5–10, hard, category):

   python baselines/clip_contrastive/evaluate.py \
       --model-dir baselines/models/clip_contrastive --data-dir data

   Use --group-sizes to restrict the sizes and --no-custom to skip the hard/category sets.

DeepMatcher (Cross-encoder)

Due to the outdated library of DeepMatcher https://github.com/anhaidgroup/deepmatcher, we copy the logic and use a different cross-encoder model.

1. Generate training pairs from train_split.csv:

   python baselines/prepare_deepmatcher_data.py

2. Train and evaluate. If no model exists at baselines/models/crossencoder_best, the cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2) is fine-tuned first and saved there. Inference then runs on all test splits (sizes 5–10, hard, category): for each group, the most similar CLIP pair is selected and classified by the cross-encoder:

   python baselines/crossencoder_baseline_all.py

LLM (self-hosted)

baselines/llm_baseline.py is an OpenAI-compatible HTTP client that asks an LLM to classify each group. It runs against any OpenAI-compatible endpoint; the recipe below self-hosts google/gemma-4-26b-a4b-it so the full matrix costs electricity instead of per-token API fees, and pins exact weights/decoding for reproducibility. The dense google/gemma-4-31b-it works too — just swap the repo id.

Serving (example: 2× ≥48 GB Ada/Hopper GPUs, e.g. L40S, via Apptainer + vLLM):

1. Smoke-test GPU passthrough, then pull the serving container:

   apptainer exec --nv docker://nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 nvidia-smi -L
   apptainer pull vllm.sif docker://vllm/vllm-openai:latest

2. Stage the weights outside the container (needs an HF token with the Gemma license accepted) and bind-mount them in:

   APPTAINERENV_HF_TOKEN="$HF_TOKEN" APPTAINERENV_HF_HOME=/hf/.hf-cache \
     apptainer exec --bind ./hf:/hf vllm.sif \
     bash -lc 'mkdir -p $HF_HOME; hf download google/gemma-4-26b-a4b-it --local-dir /hf/gemma-4-26b-a4b-it'

3. Serve. Tensor parallelism (TP=2) wedges on this stack (vLLM v1's shared-memory broadcast hangs after KV-cache allocation), so run one single-GPU FP8 server per reasoning mode instead — TP=1 + FP8 fits comfortably on one 48 GB GPU and total throughput is higher than a working TP=2 would give. NCCL_NET_PLUGIN=none + NCCL_IB_DISABLE=1 avoid a segfault from the image's Spectrum-X NCCL plugin on non-Spectrum hosts (pass them via --env on exec, not just at instance start).

   apptainer instance start --nv --writable-tmpfs --bind ./hf:/hf --bind ./cache:/root/.cache vllm.sif gemma-server
   nohup apptainer exec --env NCCL_NET_PLUGIN=none,NCCL_IB_DISABLE=1 instance://gemma-server \
     vllm serve /hf/gemma-4-26b-a4b-it --served-model-name google/gemma-4-26b-a4b-it \
     --tensor-parallel-size 1 --quantization fp8 --kv-cache-dtype fp8 --max-model-len 6144 \
     --enforce-eager --limit-mm-per-prompt '{"image":0,"video":0}' \
     --host 0.0.0.0 --port 8000 >serve.log 2>&1 &

   Start a second instance with CUDA_VISIBLE_DEVICES=1 and --port 8001 for the other GPU/mode. Wait for Uvicorn running on … in each log, then check curl -s http://localhost:8000/v1/models.

Running the baseline — one invocation per (server, mode), in parallel:

python baselines/llm_baseline.py --base-url http://localhost:8000/v1 \
    --model google/gemma-4-26b-a4b-it --modes none --n-runs 10 --workers 64 \
    --output results/llm_none.csv --cache-dir results/cache/none
python baselines/llm_baseline.py --base-url http://localhost:8001/v1 \
    --model google/gemma-4-26b-a4b-it --modes low  --n-runs 10 --workers 64 \
    --output results/llm_low.csv  --cache-dir results/cache/low

--modes none/low selects the reasoning mode (each against its own server), --n-runs 10 matches the other baselines' repeat protocol, --workers sets in-flight concurrency, and separate --output/--cache-dir keep the two processes from clashing. Predictions are flushed per call, so runs are resumable. Cap with --max-rows for a faster smoke run. Drop --quantization fp8/lower --max-model-len if you hit OOM.

Merge the per-mode CSVs when both finish:

python -c "import pandas as pd; pd.concat([pd.read_csv('results/llm_none.csv'), pd.read_csv('results/llm_low.csv')]).to_csv('results/llm_gemma-4-26b-a4b-it.csv', index=False)"

Evaluation

Compute the global F1 (mean ± std over repeats) from the per-split prediction CSVs of one or more result directories:

python baselines/evaluation/concat_and_evaluate.py results/<timestamp> [...]

For LLM predictions stored as JSONL runs (run_N/*.jsonl with {"i": <row idx>, "pred": 0|1} per line), use the JSONL variant, which joins predictions to labels in the groups_*.csv files:

python baselines/evaluation/concat_and_evaluate_jsonl.py results/<llm_results_dir> --data-dir data

The cross-encoder is deterministic and only writes repeat 0; to evaluate it with the same mean ± std protocol, generate bootstrap repeats 1–9 first:

python baselines/evaluation/generate_repeats.py results/<timestamp> --prefix ce --name size_5