llama-nemotron-embed-vl-1b-v2

Description

llama-nemotron-embed-vl-1b-v2 was developed by NVIDIA for multimodal question-answering retrieval.

The model can embed document pages in the form of:

• Images
• Text
• Combined image–text inputs

It supports document pages containing:

• Text
• Tables
• Charts
• Infographics

Core Idea

An embedding model is a key component of retrieval systems:

• Converts information into dense vector representations
• Based on a Transformer encoder architecture
• Processes text or images (queries, passages, pages)
• Outputs embeddings for similarity search

Model Family

• Part of Nemotron RAG collection
• Available on HuggingFace
• Deployable via NVIDIA NeMo Retriever (NIM)

Production Use

• Optimized for low latency + high throughput
• Enterprise-ready retrieval pipeline
• Supports customization for domain-specific applications:
  • IT assistants
  • HR systems
  • R&D assistants

License

• NVIDIA Open Model License Agreement
• Post-processing scripts: Apache 2.0
• Built with Llama 3.2 Community Model License

Deployment

• Global availability

Use Case

Best suited for:

• Multimodal QA systems
• Large-scale document retrieval
• PDF search systems (text + charts + tables)
• Vector database-based search engines

Retrieval Pipeline

Typical workflow:

1. Embed entire corpus (text or images)
2. Store embeddings in vector database
3. Embed query at runtime
4. Retrieve nearest embeddings in vector space

Release Date

• 18/12/2025

Architecture

Type

• Transformer Encoder
• Eagle VLM architecture

Components

Technical Design

• 16 transformer layers
• Embedding size: 2048
• Bi-encoder architecture
• Mean pooling over token embeddings
• Contrastive learning objective

Vision Improvements

• Eagle 2 architecture
• Nemoretriever-parse integration
• Dynamic tiling system
• Mixture of vision encoders

Improves:

• High-resolution image understanding
• Complex document parsing

Token Limits

• Max context: 10,240 tokens
• Max image tiles: 6
• Each tile = 256 tokens
• Optional thumbnail included

Output

• Dense embedding vectors
• Dimension: 2048 floats

Inputs & Outputs

Inputs

Outputs

• Float vector embeddings (2048-dim)

Installation

pip install "transformers>=4.56.0"
pip install "flash-attn>=2.6.3,<2.8" --no-build-isolation

Transformers Usage

import torch
from transformers import AutoModel
from transformers.image_utils import load_image

model_name = "nvidia/llama-nemotron-embed-vl-1b-v2"

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
    device_map="auto"
).eval()

modality = "image"

model.processor.p_max_length = 2048
model.processor.max_input_tiles = 6
model.processor.use_thumbnail = True

query = "How is AI improving robotics?"

image_paths = [
    "img1.jpg",
    "img2.jpg",
    "img3.jpg"
]

images = [load_image(p) for p in image_paths]

with torch.inference_mode():
    q_emb = model.encode_queries([query])
    d_emb = model.encode_documents(images=images)

def l2norm(x):
    return x / (x.norm(dim=-1, keepdim=True) + 1e-12)

similarity = l2norm(q_emb) @ l2norm(d_emb).T
print(similarity)

Evaluation

Vision Retrieval Benchmarks

• ViDoRe V1
• ViDoRe V2
• ViDoRe V3
• DigitalCorpora-10k
• Earnings V2

Text Retrieval Benchmarks

• BEIR
• MIRACL
• MLQA
• MLDR

Performance

Visual Retrieval (Recall@5)

Text Retrieval (Average Recall@5)

Inference

Engines

• TensorRT
• Triton
• NeMo Retriever NIM

Hardware

• NVIDIA Ampere
• Hopper
• Lovelace
• Blackwell

Ethical Considerations

• Designed for retrieval systems
• Requires system-level validation in production
• May not always retrieve correct passages

Privacy

• No known personal data used
• Dataset provenance tracked
• External data constraints apply

Safety

• Not for life-critical systems
• Retrieval accuracy not guaranteed
• Must be integrated with safeguards

Summary

This model is a multimodal embedding system designed for:

• Enterprise search
• RAG pipelines
• Document understanding
• Multimodal retrieval systems

It combines:

• Vision-language encoding
• Efficient transformer architecture
• Production-grade NVIDIA deployment stack