dft core

Purpose

Open-source core runtime, YAML contract, compatibility, and release quality. This is the engine: the compile/execute/render pipeline that turns YAML dashboard definitions into executable queries and rendered output. Covers the CLI surface, the YAML spec and its normalizer, query execution against warehouses, and the HTML/SVG rendering layer. Owns the OSS release process, versioning contract, and migration story. Adjacent to but distinct from cloud-suite (which is the hosted UX on top of core) and graph-library (which owns chart design within core's rendering layer).

Owner

• Sr Engineer Architect

Tasks by Milestone

> M0 — Prototype (3/3 completed)

A runnable prototype path exists for the YAML contract, compiler/normalizer, execution adapters, and release/versioning, with concrete artifacts that prove the flow works end-to-end in the current codebase. Core assumptions are documented, known constraints are explicit, and the team can explain what is real versus mocked without ambiguity.

• ☑ Prototype gaps and follow-on capture — Document top gaps and risks in versioning and migrations that must be addressed next.
• ☑ Prototype implementation path — Implement a runnable end-to-end prototype path for YAML contract and normalizer.
• ☑ Prototype validation and proof — Validate execution/runtime adapters with concrete proof artifacts and repeatable steps.

> M1 — 5T Internal Pilot Ready (11/11 completed)

Internal analysts can execute at least one weekly real workflow that depends on the YAML contract, compiler/normalizer, execution adapters, and release/versioning in the 5T Analytics environment, without bespoke engineering intervention for every run. Instrumentation and feedback capture are in place so failures, friction points, and adoption gaps are visible and triaged with owners.

• ☑ Remove init_sql and migrate setup patterns to query composition — Deprecate and remove init_sql/init_sql_file, then migrate setup use cases to reusable query imports (for example `_…
• ☑ Add chart renderer registry and mechanical Vega renderer — Introduce renderer-family dispatch, rebuild standard Vega-Lite translation around resolved chart semantics, and make la…
• ☑ Build chart intent and enrichment pipeline — Introduce ChartIntent, EnrichmentPatch, ResolvedChart, RenderArtifact, and the authored/enriched/config precedence flow…
• ☑ Document faces/ directory convention and dbt project setup — Fix all docs to use faces/ as the canonical dashboard directory. Add Adding Dataface to an Existing dbt Project guide t…
• ☑ Move chart output conversion into render converters — Relocate chart export conversion out of chart semantics and route SVG, PNG, and PDF conversion through render/converter…
• ☑ Rebuild geo renderer on resolved chart pipeline — Adapt geo rendering to the new intent/enrichment/resolution pipeline with shared source, join, projection, and tooltip…
• ☑ Refactor chart renderer boundaries and dispatch — The chart rendering layer still mixes orchestration, Dataface JSON serialization, SVG-native export conversion, and Veg…
• ☑ refactor: Move database/source detection from playground to core — Move database/source detection into dft core so playground and hosted surfaces share one canonical detection path.
• ☑ Deduplicate geo chart rendering and config ownership — The geo rendering code still duplicates point-map and layered-point-map behavior across geo source resolution, projecti…
• ☑ Extract shared Vega-Lite spec builder helpers — Chart-specific Vega-Lite generators still repeat the same spec assembly patterns for title handling, dimensions, toolti…
• ☑ Tighten chart enrichment and type inference boundaries — Chart enrichment and type inference still cross module boundaries in ways that blur responsibilities between render-tim…

> M2 — Internal Adoption + Design Partners (0/9 completed)

the YAML contract, compiler/normalizer, execution adapters, and release/versioning is hardened enough for regular use by multiple internal teams and initial design partners, with a predictable response loop for issues and requests. Quality expectations are documented, and prioritized improvements from real usage are actively incorporated into delivery.

• ☐ Implement YAML versioning and migrations — Add built-in schema version handling and migration workflow for dashboard YAML changes.
• ☐ Adoption hardening for internal teams — Harden YAML contract and normalizer for repeated use across multiple internal teams and first design partners.
• ☐ Design-partner feedback loop operations — Operationalize rapid feedback-to-fix loop for execution/runtime adapters with explicit decision logs.
• ☐ feat: chart decisions Phase 5 — YAML annotation overrides — Support YAML annotation overrides for chart decisions so analysts can steer defaults without patching generated specs.
• ☐ Quality standards and guardrails — Define and enforce quality standards for versioning and migrations to keep output consistent as contributors expand.
• ☐ refactor: Consolidate config system - single source of truth in YAML — Consolidate config loading into one YAML-driven source of truth to eliminate conflicting runtime settings paths.
• ☐ Return normalized Dataface format for JSON export (not Vega-Lite specs) — Make JSON export return normalized Dataface spec objects instead of raw Vega-Lite so tooling can round-trip safely.
• ☐ Add settings: attribute pattern for all chart types — Standardize a settings attribute pattern across chart types so YAML is consistent and easier to generate.
• ☐ On-demand result-set profiling in chart decisions pipeline — Expand the ColumnProfile in decisions.py _profile_columns to compute richer statistical characteristics on-demand from…

> M3 — Public Launch (0/8 completed)

Launch scope for the YAML contract, compiler/normalizer, execution adapters, and release/versioning is complete, externally explainable, and supportable: user-facing behavior is stable, documentation is publishable, and operational ownership is explicit. Remaining gaps are non-blocking, risk-assessed, and tracked as post-launch follow-up rather than unresolved launch debt.

• ☐ Launch docs and external readiness — Publish external-facing documentation and examples for execution/runtime adapters that are executable by new users.
• ☐ Launch operations and reliability readiness — Finalize operational readiness for versioning and migrations: telemetry, alerting, support ownership, and incident play…
• ☐ Public launch scope completion — Complete launch-critical scope for YAML contract and normalizer with production-safe behavior and rollback clarity.
• ☐ Publish dbt-core release — Release dbt-core package with install docs, compatibility matrix, and changelog notes.
• ☐ Declarative schema definition outside Python code — Define the Dataface YAML spec as a declarative schema document outside of Python Pydantic models — a YAML or JSON schem…
• ☐ [P3] Support raw Vega-Lite specs in Dataface YAML — Allow raw Vega-Lite blocks in Dataface YAML for advanced use cases while preserving safety and compatibility checks.
• ☐ Extensible schema with custom elements and chart types — Enable users and plugin authors to extend the Dataface schema with custom element types — custom chart types, form inpu…
• ☐ Wire up batch query prefetch before render — Add a prefetch step at the top of render that collects all visible chart queries, uses the existing batch.py infrastruc…

> M4 — v1.0 Launch (0/4 completed)

Post-launch stabilization is complete for the YAML contract, compiler/normalizer, execution adapters, and release/versioning: recurring incidents are reduced, support burden is lower, and quality gates are enforced consistently before release. The team has a repeatable operating model for maintenance, regression prevention, and measured reliability improvements.

• ☐ Regression prevention and quality gates — Add or enforce regression gates around execution/runtime adapters so release quality is sustained automatically.
• ☐ Sustainable operating model — Document and adopt sustainable operating model for versioning and migrations across support, triage, and release cadenc…
• ☐ v1.0 stability and defect burn-down — Run stability program for YAML contract and normalizer with recurring defect burn-down and reliability trend tracking.
• ☐ Async prefetch with future-based cache for progressive rendering — Extend the batch prefetch system to support non-blocking async execution. Replace the simple dict cache in Executor wit…

> M5 — v1.2 Launch (0/4 completed)

v1.2 delivers meaningful depth improvements in the YAML contract, compiler/normalizer, execution adapters, and release/versioning based on observed usage and retention signals, not just roadmap intent. Enhancements improve real customer outcomes, and release readiness is demonstrated through metrics, regression coverage, and clear migration guidance where relevant.

• ☐ Quality and performance improvements — Ship measurable quality/performance improvements in execution/runtime adapters tied to user-facing outcomes.
• ☐ v1.2 depth expansion — Deliver depth expansion in YAML contract and normalizer prioritized by observed usage and retention outcomes.
• ☐ v1.2 release and migration readiness — Prepare v1.2 release/migration readiness for versioning and migrations, including communication and upgrade guidance.
• ☐ Add optional markdown-svg raw HTML foreignObject passthrough — Add an opt-in markdown-svg configuration flag, off by default, that allows raw HTML embedded in markdown to be rendered…

> MX — Far Future (0/3 completed)

Long-horizon opportunities for the YAML contract, compiler/normalizer, execution adapters, and release/versioning are captured as concrete hypotheses with user impact, prerequisites, and evaluation criteria. Ideas are ranked by strategic value and feasibility so future investment decisions can be made quickly with less rediscovery.

• ☐ Experiment design for future bets — Design validation experiments for versioning and migrations so future bets can be tested before major investment.
• ☐ Future opportunity research — Capture long-horizon opportunities for YAML contract and normalizer with user impact and strategic fit.
• ☐ Prerequisite and dependency mapping — Map enabling prerequisites and dependencies for execution/runtime adapters to reduce future startup cost.