PureYAML/docs/CORPUS.md

Corpus Gates

PureYAML's production hardening depends on real YAML first and generated cases second. Synthetic generators are useful, but they must be anchored in documents people actually maintain.

Current State

The checked-in corpus gate is implemented. Tests/Fixtures/real-yaml-corpus.yaml is the machine-readable manifest for 114 pinned public YAML seeds copied under Tests/Fixtures/real-yaml/.

Normal swift test verifies the manifest and vendored file metadata without running the full corpus parse workload. It also runs deterministic generated properties over small valid YAML documents and malformed mutations derived from default real-corpus seeds. Run the full checked-in corpus gate explicitly when hardening parser behavior:

bash scripts/check-corpus.sh

The opt-in gate sets PUREYAML_RUN_FULL_CORPUS=1 and runs RealYAMLCorpusTests. It stream-parses every seed, validates every document, and checks parse/dump/parse stability for seeds marked as round-trippable in the manifest. A successful run writes corpus artifacts under .build/pureyaml-artifacts/.

The normal parse and parseStream paths now scan tokens lazily and compose values from parser events as those events are produced. They no longer retain a full token array or a full intermediate event array. parseEvents remains available as a debugging and compatibility surface that materializes events explicitly. This is an internal memory-reduction step; the public API still accepts a complete String, so true chunked input parsing remains tracked by #43.

RealYAMLCorpusPropertyTests runs in normal verification and covers:

• 100 deterministic generated valid YAML documents with parse/dump/parse stability;
• malformed real-seed mutations for missing mapping spaces, tab indentation, broken sequence markers, broken quoted scalars, and undefined aliases;
• structured diagnostics for every generated malformed mutation.

Run the JSON compatibility gate to prove YAML 1.2 JSON-subset behavior against Swift's JSON decoder:

bash scripts/check-json-compatibility.sh

The gate covers object, array, and scalar roots; JSON escapes; Unicode escape sequences including surrogate pairs; exponent numbers; nested values; and insignificant whitespace. It writes .build/pureyaml-artifacts/json/json-compatibility.json.

Run the Yams comparison gate separately because it intentionally uses an external dependency in a temporary package under .build/:

bash scripts/check-yams-differential.sh

This preserves Package.swift's dependency-free contract while producing .build/pureyaml-artifacts/differential/yams-comparison.json.

The manifest can mark intentional differences. The current checked-in corpus has one: a comment-only values file where PureYAML returns a single null document and Yams reports zero stream documents. The differential gate accepts only manifest-declared intentional differences.

Malformed-input diagnostics are compared separately:

bash scripts/check-yams-diagnostics.sh

This gate checks production-shaped bad YAML cases where PureYAML emits machine-readable diagnostic codes, locations, severities, and descriptions, then records the corresponding Yams parser result in .build/pureyaml-artifacts/diagnostics/yams-diagnostics.json.

Parsed-document validation is compared separately:

bash scripts/check-yams-validation.sh

This gate checks YAML that both parsers can load but that contains semantic validation issues such as duplicate mapping keys. PureYAML reports structured validation paths and reasons; the artifact records Yams' parser result and zero structured validation issues at .build/pureyaml-artifacts/validation/yams-validation.json.

Throughput comparison is also opt-in and uses the same temporary-package pattern:

bash scripts/check-throughput.sh

It measures representative checked-in corpus seeds against Yams in release mode and writes .build/pureyaml-artifacts/performance/yams-throughput.json.

PureYAML phase profiling is a separate release-mode gate:

bash scripts/check-performance-phases.sh

It times scanner, token-to-event parsing, event composition, lazy full parsing, validation, and dumping for the slow representative seeds. It writes .build/pureyaml-artifacts/performance/phase-profile.json so parser optimization work can target the dominant phase instead of guessing from total Yams comparison time. The scanner now keeps a UTF-8 cursor for byte-accurate source marks, which avoids per-character string allocation in ASCII-heavy advancement while preserving Unicode and CRLF behavior.

Round-trip expectations are tracked separately. A seed can remain in the parse, validation, and differential gates while skipping parse/dump/parse equality when the dumper does not yet preserve that document shape.

Further implementation is tracked by:

• #54, the production hardening epic.
• #55, the real-world seed, fuzz, and property-based validation gate.
• #53, Yams differential testing.
• #51, mutation-based bad-YAML validation fixtures.
• #44, humongous real-world YAML specs.

Goal

Build deterministic parser and validation gates that combine:

• pinned real-world YAML seeds,
• original-file parser and validator checks,
• deterministic valid and invalid mutations from those seeds,
• property checks for parser and dumper stability,
• differential comparison against Yams where the input is inside the shared supported subset,
• structured artifacts that explain every failure.

Every discovered bug must become a reduced fixture or exact regression test. Artifacts are useful for triage, but committed tests are the permanent truth.

Seed Sources

The first corpus should be built from public repositories with pinned commits.

Source	Seed Type	Example Paths
github/rest-api-description	Large OpenAPI 3.0 and 3.1 specs	descriptions-next/api.github.com/api.github.com.yaml, descriptions-next/api.github.com/dereferenced/api.github.com.deref.yaml, descriptions-next/ghec/ghec.yaml
APIs-guru/openapi-directory	Broad OpenAPI and Swagger corpus	APIs/1password.com/events/1.2.0/openapi.yaml, APIs/ably.io/platform/1.1.0/openapi.yaml, APIs/adyen.com/CheckoutService/70/openapi.yaml, APIs/adafruit.com/2.0.0/swagger.yaml
OAI/OpenAPI-Specification	Canonical OpenAPI examples	_archive_/schemas/v3.0/pass/petstore.yaml, _archive_/schemas/v3.0/pass/petstore-expanded.yaml, _archive_/schemas/v3.0/pass/uspto.yaml, _archive_/schemas/v3.0/schema.yaml
argoproj/argo-cd	Kubernetes, GitOps, ApplicationSet, Helm	applicationset/examples/design-doc/applicationset.yaml, applicationset/examples/git-generator-directory/cluster-addons/prometheus-operator/values.yaml
prometheus/prometheus	Good and intentionally bad app config	config/testdata/conf.good.yml, cmd/promtool/testdata/prometheus-config.good.yml, cmd/promtool/testdata/prometheus-config.bad.yml, cmd/promtool/testdata/rules_large.yml
bitnami/charts	Helm and Kubernetes chart YAML	bitnami/airflow/values.yaml, bitnami/apache/Chart.yaml, bitnami/airflow/templates/web/deployment.yaml

The corpus should include API and non-API YAML:

• OpenAPI 3.0, OpenAPI 3.1, and Swagger 2.0.
• Kubernetes manifests.
• Helm chart metadata, values files, and templates that are valid YAML.
• GitHub Actions workflows.
• Docker Compose files.
• Prometheus configs and rule files.
• Static-site, package, and application settings.
• Intentionally bad real fixtures from upstream projects when available.

Manifest

Every seed must be listed in a manifest. The manifest should include:

• stable seed ID,
• repository owner and name,
• commit SHA or release tag,
• source path,
• raw URL,
• category,
• byte size,
• line count,
• license or provenance note,
• expected parser outcome,
• expected validation outcome,
• whether the seed belongs in the fast/default corpus,
• whether the seed belongs only in the full opt-in corpus.

Huge files or files with awkward licensing should be downloaded by a corpus script rather than committed directly. Reduced fixtures that capture bugs should be committed in Tests/Fixtures/.

Corpus Sizes

Fast/default gate:

• 114 pinned public real-world YAML seeds copied into Tests/Fixtures/real-yaml/,
• normal verification checks the manifest and file metadata only,
• runs on macOS and Linux by default,
• documents Windows and WASI depth if they run a smaller subset.

Full opt-in gate:

• first opt-in gate covers the 114 checked-in seeds with parse, validation, and round-trip properties,
• default generated properties cover valid document stability and malformed real-seed mutation diagnostics,
• Yams differential gate compares parse success and stream document counts for the checked-in corpus,
• target is met with at least 100 pinned public real-world YAML seeds,
• includes at least 20 very large OpenAPI files,
• includes at least 20 non-API config files,
• includes valid and intentionally invalid upstream fixtures where available.

Mutation Coverage

Mutations should start from real seeds and cover common production mistakes:

• spacing after mapping colons,
• indentation drift,
• broken sequence markers,
• tab indentation in sensitive positions,
• malformed anchors and aliases,
• unknown aliases,
• broken quoted scalars,
• broken literal and folded block scalars,
• flow sequence and flow mapping mistakes,
• duplicate mapping keys,
• explicit key and explicit value mistakes,
• stream boundary mistakes.

Generated malformed input must assert structured validation/report behavior, not only that parsing throws.

Properties

The generated gate should include deterministic properties such as:

• successful parse and dump and parse stability where PureYAML owns both parse and dump semantics,
• no crash on malformed input,
• parser failures produce structured diagnostics when requested,
• Yams differential agreement for inputs inside the shared supported subset,
• every failure records the real seed ID and generator seed.

Every generated failure must be reproducible from a seed. Any non-trivial failure should be reduced into a stable checked-in regression fixture.

Artifacts

Corpus runs write artifacts under .build/pureyaml-artifacts/.

.build/pureyaml-artifacts/
  real-seed-manifest.json
  real-seed-summary.yaml
  generated-validation-summary.json
  generated-validation-summary.yaml
  seeds.json
  failures/
    seed-id-000123.input.yaml
    seed-id-000123.mutated.yaml
    seed-id-000123.diagnostics.json
    seed-id-000123.reduced.yaml
  differential/
    yams-comparison.json
  environment.txt

CI should upload these artifacts with the current Node 24-compatible artifact action:

- name: Upload corpus artifacts
  if: always()
  uses: actions/upload-artifact@v6
  with:
    name: pureyaml-corpus-${{ runner.os }}-${{ github.sha }}
    path: .build/pureyaml-artifacts
    retention-days: 30

Artifacts are temporary. Issues and commits must link the relevant artifact while it exists, then preserve the important case as a committed fixture.

Acceptance Checklist

• The checked-in corpus has at least 25 pinned public real-world YAML seeds.
• The full corpus has at least 100 pinned public real-world YAML seeds.
• The manifest records provenance and expected outcomes for every seed.
• The mutation gate is derived from real seeds.
• Generated failures are reproducible by seed.
• Structured validation artifacts are produced even on failure.
• Differential disagreements against Yams are fixed or documented with reduced reproducers.
• New parser bugs are reduced into permanent fixtures or exact regression tests.