UiPath/coder_eval

README preview

# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills

[![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
[![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
[![Docs](https://img.shields.io/badge/docs-coder--eval.com%2Fdocs-1f6feb.svg)](https://coder-eval.com/docs)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)

**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
**evaluating and benchmarking AI coding agents and their skills** — built for CLI
and skill builders — with sandboxing, reproducibility, and data-driven analysis.
It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
commands it actually produced. Not an "agentic coding" benchmark: it measures how
effective your CLI and skills are when used by coding agents.

Reach for it when you want to **test whether a Claude Code skill triggers**,
**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,

View full repository on GitHub →