PixelRAG

What it is

Traditional RAG parses a document down to text chunks before embedding and retrieving it — and in that parsing step, a table's structure, a chart's shape, an infographic's layout all get flattened or dropped outright. A reader model asked "what was the 2023 figure in that table" often can't answer, because the table never survived as a table.

PixelRAG skips the parsing step. It renders the page — web page, PDF, or image — into screenshot tiles, embeds those tiles directly with a vision-language embedding model, and retrieves over the images. Wikipedia's 8.28M articles ship as a pre-built, ready-to-query index; the underlying pipeline works on any document set you point it at.

How it works

Two pieces do the work: rendering documents to images instead of parsing them to text, and Qwen3-VL-Embedding — a vision-language embedding model, LoRA fine-tuned on screenshot data — which embeds each page image into a space where visual content (not just its transcribed text) is retrievable. A query, whether typed text or an image, gets embedded into the same space and matched against the tile index with FAISS (or Qdrant for larger, filterable, disk-backed deployments).

The pipeline stages

Capture is the standalone pixelshot command; everything past it runs through the pixelrag umbrella CLI, and each stage installs independently:

The train stage (fine-tuning the embedding model itself) is a separate uv project with its own pinned CUDA env — installed from inside train/, not the repo root.

Quickstart

Search the pre-built Wikipedia index — no setup

curl -X POST https://api.pixelrag.ai/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'

Render any page to screenshot tiles

pip install pixelrag
pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles

Build and serve your own index

pip install 'pixelrag[index]'

cat > pixelrag.yaml << 'EOF'
source:
  type: local
  path: ./my_docs
embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto        # cuda on Linux, mps on macOS, cpu fallback
output: ./my_index
EOF

pixelrag index build
pixelrag serve --index-dir ./my_index --port 30001

Index building works on Linux (CUDA) and macOS (Apple Silicon / MPS); device: auto picks the best backend. No GPU is required for small local jobs like a single PDF.

Give Claude eyes

The renderer also ships as a Claude Code plugin — the pixelbrowse skill. Instead of fetching raw HTML, Claude screenshots a page with pixelshot and reads the image, so it sees charts, diagrams, tables and layout the way a person does.

uv tool install pixelrag                            # puts pixelshot on PATH
claude plugin marketplace add StarTrail-org/PixelRAG
claude plugin install pixelbrowse@pixelrag-plugins

claude -p "screenshot https://news.ycombinator.com and summarize the top stories"

No MCP server, no backend — the skill just calls pixelshot (Playwright/CDP) on your own machine.

Backends

FAISS is the default for local indexes. Qdrant is the option for configurable quantization, disk-backed vectors, payload filtering, and sharing one collection across multiple PixelRAG search servers — useful once an index outgrows a single machine's memory.

Who built it

Yichuan Wang, Zhifei Li (equal contribution), Zirui Wang, Paul Teiletche, Lesheng Jin, advised by Matei Zaharia, Joseph E. Gonzalez and Sewon Min — work done at Berkeley SkyLab, BAIR and Berkeley NLP. Paper: "PixelRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation".

This page is FilePort's own write-up of the open-source StarTrail-org/PixelRAG repository — FilePort isn't affiliated with the project; the live search box above talks directly to their public hosted API.