Retirement Conductor turns a risky schema change into a verified campaign: it finds known consumers through DataHub, changes only an explicitly authorized Git/dbt target, validates with dbt, reconciles fresh evidence, and refuses the producer action until the declared evidence closes.
Retirement Conductor never interprets “nothing was returned” as “nothing depends on it.” If evidence is missing, stale, partial, ambiguous, contradictory, or natively unvalidated, retirement is refused.
DataHub knows the cross-system graph. Git knows the source. dbt knows whether the changed model works. Retirement Conductor makes those authorities satisfy one continuously reconciled completion criterion.
It can identify downstream assets, but it does not bind exact native identities, perform authorized changes, capture native validation, discover again, or stop the producer action.
It freezes evidence scope, changes a reviewable target, accepts only valid receipts, reopens on drift, and returns a machine-enforceable decision from one canonical manifest.
Flags, tickets, scores, acknowledgments, and successful API calls are context. None independently proves that a migrated consumer still behaves correctly.
Every consumer is closed by a native validation receipt, verified removal, or proved non-applicability—or remains visibly blocking.
The first supported vertical deliberately stays narrow: one warehouse column, one compatible replacement, one Git repository, native dbt validation, and DataHub as the context and reconciliation surface.
READY_TO_RETIRE means every consumer observed through the recorded sources has an acceptable disposition inside that fresh, completely paged evidence envelope. It never claims that invisible consumers cannot exist.
The system does not replace DataHub, Git, dbt, or the warehouse. It owns orchestration, durable campaign history, evidence contracts, refusal policy, and the final gate.
DataHub remains the graph and shared-context system. Git remains authoritative for repository content. dbt remains authoritative for native validation. SQLite is the single-writer campaign transaction store. The producer workflow retains the destructive credential.
Model assistance may explain consumers or propose a minimal patch. It cannot decide identity uniqueness, authorization, preconditions, state transitions, receipt acceptance, or retirement readiness.
The policy produces exactly four decisions. Operator views, the DataHub summary, the HTML report, and the producer gate all consume the same canonical manifest instead of recomputing policy independently.
READY_TO_RETIRE
All required evidence is fresh and complete within scope, every consumer is acceptably closed, and reconciliation found no new consumer.
REVIEW_REQUIRED
Technical checks are sufficient, but a declared semantic or authority judgment still requires a responsible person.
BLOCKED
A required source, permission, identity, approval, validator, freshness bound, or pagination proof is unavailable or inconclusive.
UNSAFE
A known active, failed, stale, newly discovered, unresolved, or otherwise unacceptable consumer remains.
READY_TO_RETIRE.
It rereads current state, validates provenance and digests, checks source and publication bindings, consumes one short-lived issued producer plan, and refuses replay. A prior green report or manifest is never reusable authorization.
Git/dbt is the sole automated native mutation boundary. DataHub may reveal dashboards, notebooks, pipelines, charts, or models elsewhere; those consumers stay exact blockers until independent evidence closes them.
The current engineering goal is complete. Evidence remains carefully typed as live local, fixture, replay, or analysis; an official dataset is still fixture data, and a local integration is not production coverage.
Strict schemas, valid and refusal fixtures, package smoke test, and fixture evidence forced to remain BLOCKED.
Append-only campaign events, deterministic replay, interruption recovery, approval and receipt binding, four-way policy, clock and integrity refusals.
Live Core resolved the exact fields, paged 31 consumers across seven pages, expanded one repository reference by at least 30 consumers, and verified one stable DataHub document without lifecycle mutation.
One exact DataHub-to-dbt identity, one-file review branches, bounded apply, dbt-native validation, compensation, idempotent reapply, and sandbox containment.
Fresh reconciliation, verified publication, one ready sentinel, late-consumer reopening, rich-graph refusal, and one-time producer-plan enforcement.
Canonical CLI and HTML for all four decisions, exact plan confirmation, structural redaction, responsive browser proof, 15 keyboard targets, and zero axe violations.
Four checksum-pinned official datasets, deterministic synthetic truth, direct DataHub aspect reread, 14/14 oracle matches, exact Git/dbt apply, and zero false readiness.
53 security, 47 fault, and 40 recovery tests; plan/apply separation; copied-store refusal; verified backup/restore; dependency, secret, and public-artifact scans.
Reproducible 0.2.0 package, clean Python 3.11–3.14 installs, installed-wheel live reference, upgrade/rollback, confirmed state removal, and explicit independent-operator boundary.
The benchmark uses official public inputs where they add realistic structure, then derives a deterministic field-retirement corpus with an independent oracle. Large databases stay in an ignored, content-addressed cache—not in the package or Git history.
| Official asset | Quality role | License | Observed and verified input |
|---|---|---|---|
| fiction-retail | Clean relational control and field-replacement base | CC0-1.0 | 150,000 orders; zero observed foreign-key orphans; 9f95373e…55754 |
| healthcare | Selective downstream impact from deterministic quality defects | CC0-1.0 | 1,215 negative billing, 555 null names, 832 invalid ages, 277 date swaps; 287a47c5…ec488 |
| nyc-taxi clean | Native freshness control | NYC public domain | Checksum-pinned clean control; fef53dcc…6537d |
| nyc-taxi stale | Fresh catalog metadata over stale underlying data | NYC public domain | Observed nine-day native gap; zero zero-trip mart rows; 35573de0…c9871 |
All four assets come from datahub-project/static-assets at commit a6479c691dd2a40dd89563396d9c8b2b28bee83c. The registry fixes URL, revision, license evidence, byte size, SHA-256, format, and allowed archive members. First acquisition verified 312,086,528 bytes; later runs can verify strictly offline.
The pinned NYC stale database did not match its README’s three-day and empty-load description. The benchmark preserved the discrepancy and used the queried nine-day gap rather than adjusting data to make documentation appear correct.
Generator 1.1.0, seed 20260802, medium scale: 2,500 orders joined across ten tables with stable ordering, zero clean-control integrity failures, documented distributions, byte-identical twin generation, exact fault provenance, and an oracle that does not import campaign policy.
| Truth scenario | Expected decision | Required refusal signal |
|---|---|---|
| Clean isolated replacement | READY_TO_RETIRE | No refusal; one verified dbt consumer |
| Ambiguous native/DataHub identity | BLOCKED | IDENTITY_AMBIGUOUS |
| Incomplete lineage page | BLOCKED | EVIDENCE_PAGINATION_FAILED |
| Stale native data, fresh metadata | BLOCKED | EVIDENCE_REQUIRED_SOURCE_INCOMPLETE |
| Incompatible replacement type | UNSAFE | SPEC_REPLACEMENT_INCOMPATIBLE |
| Late consumer after first reconciliation | UNSAFE | RECONCILIATION_NEW_CONSUMER |
| Null inflation | UNSAFE | VALIDATION_RECEIPT_FAILED |
| Valid-category semantic drift | UNSAFE | VALIDATION_RECEIPT_FAILED |
| Unmapped category | UNSAFE | VALIDATION_RECEIPT_FAILED |
| Healthcare branch-selective failure | UNSAFE | POLICY_CONSUMER_FAILED |
| Table-only lineage | UNSAFE | POLICY_CONSUMER_OPAQUE |
| Ownership context without native validation | UNSAFE | POLICY_CONSUMER_OPAQUE |
| Disappearing edge without closure receipt | UNSAFE | POLICY_CONSUMER_STALE |
| Rich graph with opaque consumers | UNSAFE | POLICY_CONSUMER_OPAQUE |
Read, plan, apply, validate, reconcile, publish, and producer action are distinct capabilities. Authentication alone grants no mutation authority, and the campaign process never receives the producer’s destructive credential.
Resolve one exact native identity. Bind repository, branch, commit, file, content fingerprint, DataHub URN, dbt manifest node, policy, validator, target allowlist, approval, and expiry.
Copy the dbt project into bubblewrap with no host home, source checkout, or network namespace; constrain environment and resources; treat macros, hooks, packages, and generated content as untrusted.
Reread sources and publication, verify canonical bindings, record intent before action, consume one issued plan, and refuse replay, drift, unavailable evidence, untrusted context, or a non-ready decision.
Append-oriented SQLite events are authoritative over materialized cache. The store binds one resolved path and writer identity, refuses copied state and known shared filesystems, validates canonical replay before backup, and restores only to the original authority path.
Secrets never enter specs, command arguments, manifests, public artifacts, or support packets. Raw source rows stay ignored. Public rendering structurally replaces sensitive identities before HTML generation rather than post-processing private output.
Run these commands from the repository root on the supported Linux boundary. They operate only on repository-owned disposable resources and public or generated fixtures. They do not connect to production systems or provision paid services.
Use a clean worktree. Do not substitute production endpoints, credentials, repositories, warehouses, or state paths. The positive producer proof writes a harmless local sentinel; it never deletes a warehouse field.
git status --short
uv sync --all-groups
uv run retirement-conductor --version
uv run retirement-conductor validate-spec fixtures/specs/valid.yaml
uv run retirement-conductor fixture run fixtures/specs/valid.yamlExpected fixture decision: BLOCKED with EVIDENCE_MODE_NOT_LIVE. A positive fixture must not authorize retirement.
make check
git diff --checkThe recorded final run passed Ruff, formatting, strict mypy, 183 tests, repository and link validation, a 302-file secret scan, a 56-file public-artifact review, and reproducible source/wheel builds.
# Acquires four checksum-pinned public SQLite assets on first run.
make phase06-data
# Starts only the isolated loopback Core composition.
make datahub-core-up
make phase06-benchmark
make phase06-evidence
# Re-run to prove stable semantic output and promotion.
make phase06-benchmark
make phase06-evidence
uv run python scripts/check_public_artifacts.py
uv run python scripts/check_secrets.py
git diff --check
# Stop services without deleting the isolated volumes.
make datahub-core-downInspect for 14/14 expected decisions, zero false readiness, 100% controlled-graph recall, exact one-file apply, three native fault failures, one ready sentinel, and late/rich refusal.
make datahub-core-up
make git-dbt-tool
make test-end-to-end
make phase04-evidence
make check
git diff --check
make datahub-core-downThe runner creates a unique ignored store, repository, campaign, writer, trusted run, artifact tree, and sentinel root. Expected outcomes: one READY_TO_RETIRE action, replay refusal, late-consumer reopening, and rich-graph refusal.
make package
make test-install
make test-upgrade
make test-reference-campaign
make phase08-evidence
make check
git diff --checkThis exercises reproducible archives, Python 3.11–3.14 clean installs, actionable preflight, installed-wheel operations, upgrade and backup-based rollback, copied-store refusal, confirmed state removal, and package uninstall.
jq . artifacts/public/phase06/phase06-evidence.json
jq . artifacts/public/phase06/oracle-evidence.json
jq . artifacts/public/phase06/refusal-matrix.json
jq . artifacts/public/phase07/phase07-evidence.json
jq . artifacts/public/phase08/phase08-preacceptance-evidence.json
jq . artifacts/public/phase08/operator-boundary.json
sha256sum artifacts/public/phase06/*.json
sha256sum artifacts/public/phase07/*.json
sha256sum artifacts/public/phase08/*.jsonPublic artifacts contain aggregates and digests—not source rows, planted row IDs, credentials, cache paths, private host paths, or raw native output.
Expected failures are domain results, not unexpected exceptions. The prefix identifies the failed authority and each result retains a safe next action without leaking sensitive content.
SPEC_Invalid or unsupported retirement intentEVIDENCE_Unavailable, partial, stale, or unverifiable evidenceIDENTITY_Missing, ambiguous, or inconsistent native identityAUTH_ / SCOPE_Mutation was not explicitly and exactly authorizedVALIDATION_Source-native validation failed or was inconclusiveVALIDATED.RECONCILIATION_ / POLICY_Fresh evidence reopened or prevented closureINTEGRITY_ / RUNTIME_State, provenance, or single-writer authority is not trustworthyGATE_The producer action cannot safely executeThe repository is evidence-driven. Status changes only after commands run and artifacts are inspected; no implementation should weaken a refusal merely to obtain a positive result.
AGENTS.mdGOAL.mdSTATUS.mdREADME.md and docs/PRODUCT.mdPLAN.md and the relevant phase contractdocs/REQUIREMENTS_TRACEABILITY.mdgit status --short
uv sync --all-groups
make check
make phase06-data
make datahub-core-up
make phase06-benchmark
make phase06-evidence
make phase07-evidence
make phase08-evidence
uv run python scripts/check_secrets.py
uv run python scripts/check_public_artifacts.py
git diff --check
make datahub-core-downInspect artifacts after every generator. Never promote from exit status alone, never treat an implementation failure as external access, and never convert fixture output into live evidence.
The engineering vertical is complete. Adoption and production claims are deliberately not inferred from repository work.
RC-018 remains NOT_RUN. No independent prospective operator has yet established whether the workflow is operable without the author, recurrent, valuable, adoptable, or fundable.
Executed on Linux x86_64 with Python 3.11–3.14, local single-writer SQLite, DataHub Core 1.6.0, MCP 0.6.0, Git, dbt/DuckDB, Docker, and bubblewrap. macOS, Windows, musl, shared or multi-writer state, a product container, and DataHub Cloud remain unexecuted.
Official and generated data are still fixtures. Live-local integration proves the mechanics against disposable sources, not production graph completeness, production identity behavior, warehouse deletion, broad market demand, or universal safety.
The wheel and source archive reproduce byte-for-byte and carry checksums plus a runtime SBOM, but remain unsigned. SHA-256 detects changed content; it does not prove who produced the artifact.
The producer plan is short-lived, one-time, and checked immediately before the local sentinel. A real separately privileged warehouse transaction still owns the final external race and must call the same gate inside its trusted invocation.