Modelsopenaiclip-vit-base-patch32
provideropenai /

clip-vit-base-patch32

0.175 DZD/ second

The CLIP model was developed by OpenAI to investigate the robustness of computer vision models. It uses a Vision Transformer architecture and was trained on a large dataset of image-caption pairs. The model shows promise in various computer vision tasks but also has limitations, including difficulties with fine-grained classification and potential biases in certain applications.

Public
OpenAIOpenAI
clip-vit-base-patch32
ArchitectureDense
Context Windowimage-classification

Disclaimer: This model card is adapted from the official CLIP repository.

📌 Model Details

The CLIP model was developed by researchers at OpenAI to study robustness in computer vision tasks and to evaluate the ability of models to generalize to arbitrary image classification in a zero‑shot manner.
It was not designed for direct deployment; careful study of its capabilities is required before use in applied contexts.

📅 Model Date

January 2021

🧩 Model Type

  • Image encoder: ViT‑B/32 Transformer
  • Text encoder: Masked self‑attention Transformer
  • Training objective: maximize similarity of (image, text) pairs via contrastive loss

Original implementations included:

  • ResNet image encoder variant
  • Vision Transformer variant (this repository uses the ViT version)

📚 Documents

🎯 Model Use

Intended Use

  • Research output for AI research communities.
  • Enable exploration of zero‑shot image classification.
  • Support interdisciplinary studies on the impact of multimodal models.

Primary users: AI researchers studying robustness, generalization, biases, and constraints.

Out‑of‑Scope Use Cases

  • Any deployed use case (commercial or non‑commercial).
  • Image search in unconstrained environments without thorough testing.
  • Surveillance and facial recognition tasks.
  • Non‑English language applications (model trained/evaluated only in English).

📊 Data

  • Trained on publicly available image‑caption data.
  • Sources: crawled websites + datasets like YFCC100M.
  • Data reflects internet‑connected populations (skewed toward developed nations, younger/male users).
  • Violent/adult content filtered out.
  • Dataset not released for commercial use.

Mission: Gather large‑scale, diverse internet data to test robustness and generalizability in vision tasks.

📈 Performance

Evaluated across diverse datasets including:

  • Image classification: Food101, CIFAR10/100, Birdsnap, SUN397, Stanford Cars, FGVC Aircraft, VOC2007, DTD, Oxford‑IIIT Pets, Caltech101, Flowers102, MNIST, SVHN, STL‑10
  • OCR & text tasks: IIIT5K, Hateful Memes, SST‑2
  • Video/action tasks: UCF101, Kinetics700, RareAct, Youtube‑BB, ImageNet‑Vid
  • Other benchmarks: CLEVR Counting, KITTI Distance, Flickr30, MSCOCO, ImageNet (+A, +R, Sketch, ObjectNet)

⚠️ Limitations

  • Struggles with fine‑grained classification and object counting.
  • Fairness and bias concerns (see paper).
  • Linear probe evaluations may underestimate performance.

⚖️ Bias & Fairness

  • Performance depends heavily on class design and taxonomy choices.
  • Fairface dataset evaluations showed disparities across race and gender categories.
  • Accuracy examples:
    • Gender classification: >96% overall (highest: Middle Eastern 98.4%, lowest: White 96.5%)
    • Racial classification: ~93%
    • Age classification: ~63%

These evaluations highlight potential risks and biases, not endorsements of demographic classification tasks.

💬 Feedback

Questions or comments about the model can be submitted via the official Google Form provided in the CLIP repository.