CLI Reference

All commands available in qebench, organized by the typical daily workflow.

qebench update¶

Pull the latest code, data, and dependencies from GitHub, then enrich term contexts from QuantEcon lecture repos. Run this at the start of every session to ensure you have everyone’s latest contributions and any CLI updates.

uv run qebench update

No options — it runs three steps:

1. Pull — git pull --rebase to get the latest code and data

2. Sync — uv sync to install any new or updated dependencies

3. Enrich — clone/update QuantEcon lecture repos into .cache/lectures/ and add context sentences to terms that don’t have them yet

The enrichment step scans four lecture repositories for sentences that use each term, storing up to 5 example sentences per term. These context sentences are shown during qebench translate to help you choose the right Chinese translation. The lecture repos are cached locally (shallow clones, gitignored) so subsequent runs only pull changes.

If already up to date, it tells you so. If the pull fails (e.g. you have uncommitted changes), resolve them first then try again.

qebench stats¶

Show dataset coverage, domain breakdown, and progress toward targets.

uv run qebench stats

Output includes:

• Progress bars for terms, sentences, and paragraphs vs. targets

• Domain breakdown table with entry counts

• XP leaderboard ranked by total XP (with translate/add/judge breakdown)

• Total entries summary

No options — always shows the full dataset overview.

qebench translate¶

Collect human translations. Presents English text, collects your Chinese translation and a confidence rating, then reveals the reference for learning. Every translation — including ones that differ from the reference — is valuable data for understanding translation variation.

uv run qebench translate [OPTIONS]

Your GitHub username is detected automatically via gh auth.

Examples:

# Quick 3-term session on economics
uv run qebench translate -n 3 -d economics

# Practice advanced terms
uv run qebench translate --difficulty advanced

# Default session (5 random entries)
uv run qebench translate

What’s recorded per entry:

• Your Chinese translation

• Confidence level (1–5)

• Character similarity to the reference (informational, not a grade)

• If your translation differs: the reason why (formal/informal register, regional preference, contextual, abbreviation, alternative technical term, or other)

• Optional notes for further explanation

Divergent translations are valuable — they help us understand cultural nuance and variation.

For terms that have context sentences (populated by qebench update), a random example sentence from a QuantEcon lecture is shown alongside the term. This helps you understand how the term is used in practice and choose the most appropriate Chinese translation.

Each completed entry earns 10 XP. A cli_version field is automatically saved with every record for future schema migration.

qebench add¶

Contribute new terms, sentences, or paragraphs to the dataset through interactive prompts.

uv run qebench add

No options — the command walks you through the process:

1. Choose entry type — term, sentence, or paragraph

2. Fill in fields — English text, Chinese translation, domain, difficulty, etc.

3. Preview — see a summary before saving

4. Confirm — save to the appropriate domain JSON file

5. Continue? — option to add another entry

Each contributed entry earns 15 XP. A cli_version field is automatically saved with every entry.

qebench judge¶

Judge anonymous translations head-to-head. Shows two translations of the same source text, you rate each on accuracy and fluency, then pick a winner. Results update Elo ratings for the models.

uv run qebench judge                       # Default: 10 rounds
uv run qebench judge -n 5                  # Quick 5-round session
uv run qebench judge -d economics          # Filter to economics entries

Options¶

Prerequisites¶

Model outputs must exist in results/model-outputs/. Generate them with qebench run first.

How It Works¶

1. Entries are paired with model translations from results/model-outputs/

2. Two translations are shown anonymously as A and B

3. You rate each on accuracy (0–5) and fluency (0–5)

4. You pick a winner (A, B, tie, or neither)

5. Elo ratings are updated; results saved to results/judgments/

6. The reveal panel shows model labels, Elo ratings, reference overlap, glossary compliance, and formatting scores (fullwidth punctuation %, directive balance)

Pick Tie if both translations are equally good. Pick Neither if both translations are poor and neither is acceptable.

If two models have translated the same entry, they’re paired directly. If only one model has output, it’s paired against the human reference. Identical translation pairs are automatically skipped.

Each judgment earns 5 XP.

qebench submit¶

Pull latest changes, commit your data and results, and push to GitHub. This is the primary way to share your contributions.

uv run qebench submit

No options — it handles the full git workflow:

1. Pull — git pull --rebase to get latest changes

2. Stage — adds data/ and results/ directories

3. Commit — creates a commit attributed to your GitHub username

4. Push — pushes to main, which triggers a dashboard rebuild

If there are no local changes in data/ or results/, it exits early.

qebench doctor¶

Run preflight checks to verify your environment is set up correctly.

uv run qebench doctor

Checks performed:

• GitHub CLI (gh) installed

• GitHub authentication configured

• Git installed and inside a repo

• Remote origin configured

• config.yaml found

• Dataset has entries

• uv package manager available

Run this once after initial setup, or whenever something seems wrong.

qebench validate¶

Validate all dataset JSON files against the Pydantic schemas. Useful for checking your contributed entries before submitting.

uv run qebench validate            # schema errors fail; alignment problems warn
uv run qebench validate --strict   # alignment problems fail too

Options¶

Checks every file in data/terms/, data/sentences/, and data/paragraphs/ against the corresponding model (Term, Sentence, Paragraph). Reports all validation errors with file names and entry IDs, then exits non-zero if any were found.

En/zh alignment¶

Sentence and paragraph entries are additionally checked for whether their zh really is a translation of their en — a misaligned reference is worse than a missing one, since qebench judge scores every judgment against it. The check compares three signals that survive a faithful translation: the inline $...$ math spans, the targets of {doc}/{eq}/{ref}-style roles, and the zh/en length ratio. Terms are skipped — a headword has no markers and no comparable length.

Findings are reported as warnings and do not fail the command, because the check is a heuristic — a legitimately terse translation can trip the length ratio. They are printed in their own block, with the command to inspect the flagged text:

uv run python scripts/audit_alignment.py --show-text

--strict promotes those warnings to errors, so any finding exits non-zero. Use it when you want the same verdict CI gives you: the CI workflow runs qebench validate --strict on every push to main and every pull request, so an alignment finding blocks a merge even though a local qebench validate lets it through.

qebench run¶

Batch translate dataset entries using an LLM provider. Results are saved to results/model-outputs/ as JSONL files.

uv run qebench run                             # Default: claude, all terms
uv run qebench run --provider openai            # Use OpenAI
uv run qebench run --model gpt-5.4-mini         # Override model
uv run qebench run --prompt academic            # Use academic prompt template
uv run qebench run --prompt action-new          # MyST-aware prompt with glossary
uv run qebench run --type sentences --domain economics  # Filter entries
uv run qebench run --count 10 --dry-run         # Preview without API calls
uv run qebench run --prompt action-new --no-cache  # Bill the glossary on every call

Options¶

Prompt caching¶

Every template ends with the {text} placeholder, so everything before it is the same for each entry in a run. qebench run sends that prefix as its own cacheable block, and the entry itself as a second block — the prompt text is unchanged, only its packaging.

This matters for action-new, whose prefix carries the whole glossary: around 5,200 of the roughly 5,250 tokens per term. Cached, that prefix is written once per distinct prefix and read back at a tenth of the input rate for every entry after it.

Two details are worth knowing:

• action-new renders one prefix per domain, because the template interpolates {domain} on line 3, ahead of the glossary. A terms run therefore writes 15 cache entries rather than one. qebench run translates one entry per domain before fanning out the rest, so the writes happen once instead of once per worker.

• A domain with a single entry is not cached at all. A cache write costs 1.25x the input rate, so an entry nothing reads back would only cost more. Three of the 17 paragraphs are the only entry in their domain.

--no-cache sends the original single-block prompt, which is what the runs recorded in NOTES.md before v0.7.0 used. OpenAI caches long prefixes on its own with nothing to declare on the request, so there the flag only controls whether a run warms the cache before fanning out.

Prompt Templates¶

Four prompt templates are available:

The action-new template uses the {glossary} placeholder, which is automatically populated from action-translation’s glossary (fetched from GitHub and cached locally in .cache/glossary.json). See Glossary & Prompt Templates Tutorial.

Prerequisites¶

Install LLM dependencies:

uv sync --extra llm

Set your API key via environment variable (ANTHROPIC_API_KEY or OPENAI_API_KEY).

qebench export¶

Export dataset statistics and results to JSON files for the dashboard website.

uv run qebench export

Writes 8 JSON files to docs/_static/dashboard/data/:

• coverage.json — terms/sentences/paragraphs vs. targets

• domains.json — per-domain entry counts

• difficulty.json — basic/intermediate/advanced distribution

• leaderboard.json — XP rankings across users

• activity.json — recent translation attempts

• samples.json — sample terms for the browse section

• ratings.json — model Elo ratings and mean judge scores

• models.json — formatting fidelity and glossary compliance per model, prompt and entry type

This is run automatically by CI when changes are pushed.

Where the model comparison comes from¶

ratings.json needs a human in the loop — a model only appears once someone has judged it. models.json is computed from the committed run files in results/model-outputs/, so it covers every model and prompt that has ever been run, judged or not. The two are complementary rather than redundant.

Glossary compliance is scored against the upstream action-translation glossary, not the dataset’s own key_terms field, which is empty for every committed entry and would score a vacuous 100%. A record the glossary has nothing to say about is excluded rather than counted as compliant, so the number of records actually scored travels with the mean.

Two caveats worth carrying into any reading of it. The check is plain containment, so a translation that happens to contain the expected characters counts as compliant even when the text around it is wrong — treat it as an upper bound. And action-new injects the glossary into its own prompt, so 100% there means the model followed an instruction it was given, not that the translation is better than one scoring 78%.

Where model ratings come from¶

ratings.json is recomputed from the committed judgment logs in results/judgments/*.jsonl every time export runs — those logs are the source of truth, not results/elo.json. That local file is gitignored, so before this existed a judge’s ratings stayed on their own machine and never reached the dashboard. Now anyone who commits judgments sees them counted.

Ratings are reported at two granularities, because judgment records name their competitors two different ways. Records from v0.3 onwards use model:prompt, but v0.2-era records name a bare model, and there is no way to recover which prompt those used. Rather than guess:

Quote by_model for model-selection guidance and by_model_prompt when comparing prompt templates. The judgments field reports how many records were behind each number — worth reading before treating a ranking as settled.

Judgments against human-reference are excluded from ratings, matching qebench judge: the dataset’s own translation is not a competitor. A model judged only against the reference therefore appears in scores_by_model with no entry in either Elo table — claude-sonnet-4-20250514 is in exactly that position today, holding the best mean score on the strength of five judgments that rated no competitor. Read a score without a rating beside it as un-ranked, not as a winner.

Accuracy and fluency are normalised to the 0–5 scale, since judgments before v0.4.0 used 1–10. Which scale a record used is read from its own scores rather than its cli_version, because that field stamps the last released version and not the code that was running — this repo’s log already contains v0.4.0-only consensus records stamped 0.3.1, and trusting the stamp there would rescale a top score of 5 down to 2.22.

XP System¶

Actions earn experience points tracked per user:

XP is stored in results/xp/{username}.json and shown at the end of each session.