CIRR Dataset

For humans the advantage of a bi-modal query is clear: some concepts and attributes are more succinctly described visually, others through language. By cross-referencing the two modalities, a reference image can capture the general gist of a scene, while the text can specify finer details.

We identify a major challenge of this task as the inherent ambiguity in knowing what information is important (typically one object of interest in the scene) and what can be ignored (e.g., the background and other irrelevant objects).

CIRR contains annotations, raw images, and optional pre-extracted image features. Its organization follows Fashion-IQ. Start with the annotations, then add the images or features you need under data/cirr/.

Annotations

Clone the cirr_dataset branch into a local data/cirr/ folder:

# create a `data` folder at your desired location
mkdir data
cd data

# clone the cirr_dataset branch to the local data/cirr folder
git clone -b cirr_dataset git@github.com:Cuberick-Orion/CIRR.git cirr

The dataset repository contains the annotations. See the directory structure and file reference below for their organization and fields.

Paper correction: Table 2 should report 4,181 validation pairs, rather than 4,184.

Raw images

CIRR uses images from NLVR2. To obtain them:

1. Follow the NLVR2 direct-image-download instructions and submit the form agreeing to its Terms of Service.
2. If the NLVR2 team does not respond, email us.
3. In your email, explicitly confirm that you submitted the NLVR2 form and agreed to its terms.

Use the direct image archive. Downloading individual images by URL is not recommended: many links are broken, and those downloads lack the required subfolder structure in train/. Preserve the original filenames and folders when extracting the images.

Raw image download guidance

Pre-extracted image features

Features are optional. Each supplied ZIP contains individual .pkl files; extract it into data/cirr/, retaining the directory structure.

Directory structure

In filenames, VER is the dataset version and SPLIT is train, val, or test1.

Keep the NLVR2 image filenames and numeric training subfolders. These folder numbers carry no special meaning in CIRR. Both feature directories follow the same subfolder structure as img_raw/.

The raw-image validation folder is named dev/, while its annotation files use val.

data/cirr/
├── captions/
│   ├── cap.VER.test1.json
│   ├── cap.VER.train.json
│   └── cap.VER.val.json
├── captions_ext/
│   ├── cap.ext.VER.test1.json
│   ├── cap.ext.VER.train.json
│   └── cap.ext.VER.val.json
├── image_splits/
│   ├── split.VER.test1.json
│   ├── split.VER.train.json
│   └── split.VER.val.json
├── img_raw/
│   ├── train/
│   │   ├── 0/<image_id>.png
│   │   ├── 1/<image_id>.png
│   │   ├── 2/<image_id>.png
│   │   └── ...
│   ├── dev/<image_id>.png
│   └── test1/<image_id>.png
├── img_feat_res152/
└── img_feat_frcnn/

{
  "pairid": 12063,
  "reference": "test1-147-1-img1",
  "target_hard": "test1-83-0-img1",
  "target_soft": {
    "test1-83-0-img1": 1.0
  },
  "caption": "remove all but one dog and add a woman hugging   it",
  "img_set": {
    "id": 1,
    "members": [
      "test1-147-1-img1",
      "test1-1001-2-img0",
      "test1-83-1-img1",
      "test1-359-0-img1",
      "test1-906-0-img1",
      "test1-83-0-img1"
    ],
    "reference_rank": 3,
    "target_rank": 4
  }
}

{
  "pairid": 12063,
  "reference": "test1-147-1-img1",
  "target_hard": "test1-83-0-img1",
  "caption_extend": {
    "0": "being a photo of dogs",
    "1": "add a big dog",
    "2": "more focused on the hugging",
    "3": "background should contain grass"
  }
}

{
  "test1-147-1-img1": "./test1/test1-147-1-img1.png"
}

{
  "train-11041-2-img0": "./train/34/train-11041-2-img0.png"
}

image_id = "test1-147-1-img1.png"
feature_filename = image_id.replace(".png", ".pkl")
# test1-147-1-img1.pkl

The test-split ground truth is kept private. Submit your model’s prediction JSON files to the evaluation server to obtain test-split scores. Validation-split ground truth remains publicly available for local development and evaluation.

Sign in with GitHub, choose a prediction file, and complete hCaptcha verification. The How-To guide covers file formats, submission limits, example files, and results.

If the site is unavailable, please contact us.

Concurrently, we release the code and pre-trained models for our method Composed Image Retrieval using Pretrained LANguage Transformers (CIRPLANT). Together with the dataset, we believe this work will inspire further research on this task on a finer-grain level.

Our code is in PyTorch, and is based on PyTorch Lightning.

The paper, Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models, is available as a PDF and on arXiv.

• Sept. 2026: The test-split evaluation server returned online after a three-day outage and migration to Cloudflare.
• Oct. 2024: Added guidance for researchers having trouble obtaining raw images from NLVR2.
• Jun. 2024: Updated the download links.
• Aug. 2021: Released the dataset and code, and opened the test-split evaluation server.

• We have licensed the code and annotations of CIRR under the MIT License. Please refer to the LICENSE file for details.
• Following NLVR2 Licensing, we do not license the images used in CIRR, as we do not hold the copyright to them.
• The images used in CIRR are sourced from the NLVR2 dataset. Users are bound by its Terms of Service.

Please cite our paper if it helps your research:

@InProceedings{Liu_2021_ICCV,
    author    = {Liu, Zheyuan and Rodriguez-Opazo, Cristian and Teney, Damien and Gould, Stephen},
    title     = {Image Retrieval on Real-Life Images With Pre-Trained Vision-and-Language Models},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {2125-2134}
}

If you have any questions regarding our dataset, model, or publication, please create an issue in the project repository, or email us.