Entry standard · v1
What it takes to enter a registry.
One standard, three registries. The common fields apply to everything; each registry adds the fields its kind of work needs. Missing fields block a release.
Common fields
Required for every entry.
- Urban question
urban_question - The city problem the work addresses, and who is meant to use the result.
- Data provenance
data_provenance - Sources, geography, time span, licence, and processing steps for every dataset used.
- Evaluation
evaluation - Tasks, metrics, baselines, and protocol, with results reported as released.
- Failure modes
failure_modes - Known limitations, observed errors, and conditions under which the result should not be trusted.
- Geographic validity
geographic_validity - Cities tested, and whether claims are local or supported by cross-city evidence.
- Artifacts & citation
versionlicenselinkscitation - Code, configurations, weights or task files, a version, a licence, and how to cite the entry.
Registry 01
Model fields.
Urban question
urban_questionThe city problem the work addresses, and who is meant to use the result.Intended task
intended_taskThe prediction, representation, or generation task, and the urban decision it is meant to inform.Inputs & modalities
inputs_modalitiesImagery, maps, mobility traces, text, tabular records, or sensor streams, with resolution and preprocessing.Training data
data_provenanceSources, geography, time span, licence, sampling, and any data that was excluded and why.Evaluation
evaluationHeld-out tasks, metrics, baselines, and uncertainty estimates, with the evaluation code released.Failure modes
failure_modesObserved errors, subgroup gaps, distribution shifts, and conditions under which outputs should not be used.Geographic validity
geographic_validityCities trained and tested on, and evidence for or against transfer to other places.Usage constraints
usage_constraintsIntended users, prohibited uses, and requirements for human review.Artifacts & citation
version license links citationCode, configuration, weights or access route, version, licence, and citation.Registry 02
Agent fields.
Urban question
urban_questionThe city problem the work addresses, and who is meant to use the result.Task & workflow
task_workflowThe research or planning task, the steps the agent performs, and where the workflow starts and stops.Tools & data access
data_provenanceModels, datasets, APIs, and documents the agent can call or read, with their licences.Decision boundaries
decision_boundariesWhat the agent may decide alone, what requires a human, and how its outputs are reviewed.Evaluation protocol
evaluationSuccess criteria, reliability across runs, comparison with human performance, and reproducibility.Failure modes
failure_modesObserved errors, fabrication risks, bias in retrieved evidence, and misuse scenarios.Traceability
traceabilityLogs, intermediate outputs, and citations the agent produces so that results can be audited.Geographic validity
geographic_validityCities tested, and whether claims are local or supported by cross-city evidence.Usage constraints
usage_constraintsIntended users, settings where the agent must not be used, and cost or rate considerations.Artifacts & citation
version license links citationCode, prompts or policies, configuration, version, licence, and citation.Registry 03
Benchmark fields.
Research question
urban_questionThe urban capability the task measures, and why that capability matters for a decision.Task definition
task_definitionInputs, expected outputs, and the exact instructions or prompts given to a system under test.Data
data_provenanceSource, geography, time span, licence, splits, and how test data is protected from leakage.Metrics & criteria
metricsScoring rules, what counts as success, and how uncertainty or disagreement is reported.Baselines
baselinesReference systems, including simple and human baselines where they exist, with their scores.Task validation
evaluationHow the task itself was checked: annotation agreement, leakage tests, and whether scores track the capability it claims to measure.Known limitations
failure_modesWays the benchmark can be gamed or misread, and capabilities it does not measure.Generalization protocol
geographic_validityHow performance is tested across cities, periods, or populations beyond the development data.Submission rules
submission_rulesHow results are submitted, what must be disclosed, and how evaluation code is run.Versioning & errata
versioning_errataVersion history, corrections to data or scoring, and how earlier results are handled.Artifacts & citation
version license links citationTask files, evaluation code, version, licence, and citation.Release checklist
Before an entry goes live.
Card completeEvery required field is filled. “To follow” is not an accepted value.
Evaluation reproducibleEvaluation code runs from the released artifacts and reproduces the reported numbers.
Licences checkedData and code licences permit the release, and restricted data is described rather than redistributed.
Version assignedThe entry carries a version, a release date, and a citation; later changes get a new version.
Status & versioning
How entries change over time.
draftCard in preparation. Stored in the repository but not shown on the registry pages.
releasedCard complete and artifacts public. Shown on the registry page with its version.
deprecatedSuperseded or withdrawn. Remains listed for the record, with a pointer to its replacement where one exists.
VersionsSemantic versions. A change to data, scoring, weights, or task definition increments the version; results are always reported against a version.
Out of scope
What is not accepted.
Demos without evaluationInterfaces or outputs with no stated task, metric, or baseline.
Unverifiable claimsResults on data that cannot be described well enough for an independent check.
Undocumented transferClaims of generality without evidence beyond the development city.
Restricted materialData, weights, or documents the submitter does not have the right to release.
Propose an entry
Open an issue with the card attached.
Proposals are reviewed against this standard in the public repository. Corrections to existing entries follow the same route.