Documentation

Dashboard — Live leaderboard, dataset coverage, and activity feed.

Model Output Notes — What the committed model runs found: cost, formatting fidelity, and prompt effects, regenerable with scripts/analyze_runs.py. Two results to know before you spend anything on a run of your own: under action-basic, Haiku returns documentation instead of a translation for 42.7% of terms (134/314), and action-new re-sends the whole glossary on every call, which makes it 57× the cost of default over the same 314 terms (Sonnet 4.6: $4.9973 vs $0.0872).

User Guide (user/)¶

Documentation for RAs and contributors using qebench.

• Getting Started — Install, configure, run your first command

• Working with uv — Python package manager intro and common workflows

• CLI Reference — All commands, options, and examples

• Tutorials

  • Your First Translation Session — Walk through qebench translate

  • Contributing Entries — Grow the benchmark with qebench add

  • Updating Datasets — Keep data current with qebench update

  • Judging Translations — Compare translations and build Elo ratings with qebench judge

  • Running LLM Benchmarks — Batch translate with Claude or OpenAI via qebench run

  • Glossary & Prompt Templates — Use action-translation prompts with glossary injection

Developer Guide (developer/)¶

Documentation for developers building and maintaining qebench.

• Architecture — Module structure, data flow, design decisions

• Data Models — Pydantic schemas, JSON format, validation

• Contributing — Development workflow, testing, code style

• Seeding from Lectures — Extract aligned sentence/paragraph pairs from lecture repos