Few retrieval benchmarks exist for multimodal embeddings. The most famous ones for English are "MS-COCO" and "Flickr30k". Evaluating uform-vl-english model, one can expect the following numbers for search quality.

For multilingual benchmarks, we've created the unum-cloud/coco-sm repository². Evaluating the unum-cloud/uform-vl-multilingual-v2 model, one can expect the following metrics for text-to-image search, compared against xlm-roberta-base-ViT-B-32 OpenCLIP model.

All languages:

For captioning evaluation we measure CLIPScore and RefCLIPScore³.

Results for VQAv2 evaluation.

¹ Train split was in training data.
² Lacking a broad enough evaluation dataset, we translated the COCO Karpathy test split with multiple public and proprietary translation services, averaging the scores across all sets, and breaking them down in the bottom section.
³ We used apple/DFN5B-CLIP-ViT-H-14-378 CLIP model.

UForm comes pre-packaged with speed benchmarks for the models.

$ python python/scripts/bench_encoders.py --help
usage: bench_encoders.py [-h] [--filter-out FILTER_OUT] [--batch-size BATCH_SIZE]

options:
  -h, --help            show this help message and exit
  --filter-out FILTER_OUT
                        Filter out models, backends, or devices with a Regular Expression.
  --batch-size BATCH_SIZE
                        Batch size for the benchmark. Batch size 1 measures latency. Large batch sizes may not fit on every GPU.

On Nvidia B200, the results may look like:

uv run python python/scripts/bench_encoders.py --batch-size 2048 --gpu --torch

On the 160-core dual-socket Intel Emerald Rapids CPU-only setup, the results may look like:

uv run python python/scripts/bench_encoders.py --batch-size 128 --cpu --torch --onnx

$ python python/scripts/bench_decoders.py --help
usage: bench_decoders.py [-h] [--filter-out FILTER_OUT] [--batch-size BATCH_SIZE]

options:
  -h, --help            show this help message and exit
  --batch-size BATCH_SIZE
                        Batch size for the benchmark. Batch size 1 measures latency. Large batch sizes may not fit on every GPU.
  --max-length MAX_LENGTH
                        Maximum length of the generated text in tokens.

On Nvidia H100 GPU, the following performance is expected on text token generation using float16, equivalent PyTorch settings, and greedy decoding.

On Nvidia RTX 3090, the following performance is expected on text token generation using float16, equivalent PyTorch settings, and greedy decoding.