Registry 03 / Benchmarks
Grounded evaluation.
Open tasks that test generalization, fairness, uncertainty, and practical usefulness. A benchmark enters this registry with its data, metrics, baselines, and submission rules published.
Entries
Released benchmarks.
No public entries yet.
Benchmarks are listed here once a release meets the standard. Follow the repository for releases, or propose a task through an issue.
Benchmark card
What a benchmark entry documents.
Every benchmark entry answers the following. The name in code is the key used in the registry data and in the proposal form.
- Research question
urban_question - The urban capability the task measures, and why that capability matters for a decision.
- Task definition
task_definition - Inputs, expected outputs, and the exact instructions or prompts given to a system under test.
- Data
data_provenance - Source, geography, time span, licence, splits, and how test data is protected from leakage.
- Metrics & criteria
metrics - Scoring rules, what counts as success, and how uncertainty or disagreement is reported.
- Baselines
baselines - Reference systems, including simple and human baselines where they exist, with their scores.
- Task validation
evaluation - How the task itself was checked: annotation agreement, leakage tests, and whether scores track the capability it claims to measure.
- Known limitations
failure_modes - Ways the benchmark can be gamed or misread, and capabilities it does not measure.
- Generalization protocol
geographic_validity - How performance is tested across cities, periods, or populations beyond the development data.
- Submission rules
submission_rules - How results are submitted, what must be disclosed, and how evaluation code is run.
- Versioning & errata
versioning_errata - Version history, corrections to data or scoring, and how earlier results are handled.
- Artifacts & citation
versionlicenselinkscitation - Task files, evaluation code, version, licence, and citation.