#DATA — convert evidence into controlled datasets

DATA-001 — Intake and applicable-file decision

Start only from PERSISTENTLY SAVED AND VERIFIED versions of the current Project Control Packet, reviewed research records, acceptance matrix, and declared delivery authorization. Inventory the candidate inputs by stable ID and version; reject excerpts presented as complete files, unreviewed advisory output presented as evidence, inherited permissions, and unresolved instruction conflicts.

Select the smallest set of basenames whose row grains are required by the project. The catalog is an allowlist, not a requirement to manufacture empty tables or fragment every attribute. For each selected basename, record its purpose, “one row represents” definition, fields, logical types, primary and foreign keys, relationships, null rules, ordering, provenance, applicability, review owner, and release dependency. Produce .csv, .json, and a complete readable .md view from one logical record set; produce no form for an inapplicable basename.

DATA-002 — Release-view catalog

#DATA owns the following 22 unique basenames and their selection rules. #TEMPLATE supplies fillable records but does not create a second catalog. CSV and JSON remain the canonical machine pair; Markdown is a complete readable companion for the same records rather than a second schema.

DATA-003 — Grain, keys, and cardinality

Freeze each selected file's grain before defining fields. Primary keys are nonnull and unique; composite uniqueness rules are explicit; foreign keys name one target file and field. A nullable foreign key must have a companion reason or status that explains why the relationship is absent. Do not store IDs from unrelated tables in one undeclared polymorphic field.

State and validate every one-to-one, one-to-many, and many-to-many relation. Use bridge records for categories and entity relations rather than packed lists or repeated child columns. Temporal relationships carry a precisely defined start, end, as-of value, or declared unknown precision as applicable. Shared sites, multiple operators, renames, splits, mergers, and successors must not be forced into permanent one-to-many assumptions.

Place a value only at the grain it measures. Keep a parent total at the parent grain, and never copy it onto every child. When a Fact is not entity-centered, declare a stable subject key or a dedicated observation family rather than overloading entity_id.

DATA-004 — Stable identity and entity resolution

Use nonsemantic string identifiers in both formats, preserve leading zeros, and never recycle a retired ID. Keep raw labels and external identifiers in aliases; do not merge solely on case-folded or normalized names. Record the match method, evidence, confidence, resolution status, reviewer type, and any superseding or split identity.

Distinguish legal organization, fiscal or reporting unit, operator, site, program, service area, geography, and other project-defined subject types. Low-confidence matches and material duplicates remain in review_queue until accepted. An unresolved identity may retain a provisional stable key without silently becoming an accepted entity.

DATA-005 — Provenance layers and #FACT storage

Keep these layers separately addressable: source identity; enduring document; document version; URL occurrence; access event; source assertion; observation; claim; and evidence link. One document may have many versions, one version may occur at many URLs, and one URL may have many observed events. Mirrors, redirects, archives, and canonical locations are not collapsed without evidence.

Use GENERATED, DISCOVERED, RETURNED, OPENED, REVIEWED, CITED, and BLOCKED only for the corresponding observed URL or access state. A generated URL is not an executed query; a returned link is not opened content; an opened document is not automatically reviewed; and review does not by itself make a source cited. Unknown historical publication, discovery, access, or review times remain null with a reason instead of receiving the current timestamp.

#FACT remains the human-facing atomic research concept. Its released storage is an observation or claim linked to its source_assertions and exact evidence_links; it is not a second denormalized fact table. Evidence links use explicit nullable source_assertion_id, observation_id, and claim_id columns with exactly one target set, plus a document_version_id, locator, and evidence role.

DATA-006 — Measurements, derivation, and missingness

Store OBSERVED, DERIVED, and INFERRED as derivation classes separate from verification status, confidence, and reviewer type. Preserve a source value, label, unit, scale, currency, denominator, scope, basis, and precision before adding any standardized value. A derived value identifies its inputs and transformation; an inference states its rationale, uncertainty, and required review.

Null means no value is present. Declare controlled missing reasons, including not_reported, not_applicable, not_available, inaccessible_source, outside_scope, suppressed, conflicting_sources, unresolved_entity, not_yet_reviewed, and unknown_historical_time. Never substitute zero, false, an empty string, a sentinel, or a source token such as “N/A” for null; retain meaningful source tokens in a separate raw-value field.

Represent event/effective, period start/end, as-of, publication/update, access, analysis-cutoff, and release time separately. Record date precision and fiscal, academic, calendar, quarter, rolling, or custom period type without inventing a month or day. Document decimal scale, rounding, comparison method, and whether the logical value is a decimal number or a precision-preserving numeric string.

For mixed-age, mixed-period, mixed-scope, or joint-service values, retain eligible and ineligible components, allocation driver, formula, residual, and conditional bounds. Do not invent a central estimate from an unknown share. For interentity payments, preserve payer, nullable recipient plus reason, amount, period, accounting basis, reporting perimeter, reversals or netting, and recipient-resolution status. Keep consolidation, funding flow, final-service expenditure, and geographic attribution as separate views.

DATA-007 — CSV, JSON, and Markdown semantic parity

Every selected basename produces <basename>.csv, <basename>.json, and <basename>.md from one logical record set. CSV contains one header and one logical row per record; JSON contains one top-level array with one flat object per CSV row; Markdown contains every row and field in a readable table or repeated-record structure without truncation. All forms use identical field names, field sets, primary-key values, logical values, ordering, and null meanings. Every JSON object includes nullable keys with null; CSV uses an empty cell only where the dictionary declares null; Markdown renders an explicit declared null token plus its companion reason.

The data dictionary declares each field as string, identifier, integer, decimal, precision-preserving numeric string, boolean, calendar date, partial date/reporting period, timestamp, URL, or enum. IDs and postal codes are strings. Booleans are lowercase true/false in CSV and native booleans in JSON. Dates and timestamps use declared canonical forms. Decimal comparison is performed with decimal semantics, never binary-float approximation; scale, rounding, and raw precision are documented.

Use stable deterministic row ordering in all three files for review, while treating keys rather than position as identity. CSV header order and JSON key order follow the dictionary; Markdown field order does the same. Document Unicode normalization used for matching and preserve meaningful original text.

DATA-008 — Validation, acceptance scenarios, and review

Before release, record structural, parity, primary/composite-key, foreign-key, cardinality, deduplication, hierarchy, period, unit, provenance, evidence, missingness, count-reconciliation, and source-lineage results. Spot-check every high-impact row, inference, correction, manual transcription, and low-confidence identity plus a declared sample across sources, periods, and entity types.

Model-assisted review and human review are separate reviewer types with their own identity, time, scope, and evidence. A model cannot accept a material exception merely by labeling it reviewed. Each result is PASS, FAIL, WARNING, NOT_APPLICABLE, BLOCKED, or ACCEPTED_EXCEPTION; only the designated human may accept an exception classified as material or blocking.

The release validation set must exercise these topic-neutral cases:

DATA-009 — Immutable release, correction, and refresh

Assign a stable dataset ID and unique release ID. The release record and manifest state cutoffs, dictionary and format versions, file hashes, row counts, status, known gaps, validation outcome, and acceptance identity. A candidate becomes accepted only after blocking checks pass and the authorized human approves the packet and documented exceptions.

Never mutate or overwrite an accepted release. A correction or refresh creates a new immutable release, preserves prior artifacts, records affected file, row, field, old and new logical values, reason, evidence, decision owner, and first corrected release, and marks superseded assertions without erasing their history. release_changes classifies rows as added, changed, superseded, unchanged, removed from scope, or unresolved.

A refresh rechecks scope, cutoffs, dictionary compatibility, inaccessible sources, document versions, URL events, aliases, entity relationships, and all affected validation gates. Persist and verify corrections before regenerating downstream views. Publish only the selected three-format views and approved documentation through the configured destination, register them in the recovery manifest and Research archive, then retain each publication receipt and readback result separately from dataset acceptance.