PureYAML/docs/CORPUS.md

6.6 KiB

Corpus Gates

PureYAML's production hardening depends on real YAML first and generated cases second. Synthetic generators are useful, but they must be anchored in documents people actually maintain.

Current State

The corpus gate is designed but not implemented yet. Implementation is tracked by:

  • #54, the production hardening epic.
  • #55, the real-world seed, fuzz, and property-based validation gate.
  • #53, Yams differential testing.
  • #51, mutation-based bad-YAML validation fixtures.
  • #44, humongous real-world YAML specs.

Goal

Build deterministic parser and validation gates that combine:

  • pinned real-world YAML seeds,
  • original-file parser and validator checks,
  • deterministic valid and invalid mutations from those seeds,
  • property checks for parser and dumper stability,
  • differential comparison against Yams where the input is inside the shared supported subset,
  • structured artifacts that explain every failure.

Every discovered bug must become a reduced fixture or exact regression test. Artifacts are useful for triage, but committed tests are the permanent truth.

Seed Sources

The first corpus should be built from public repositories with pinned commits.

Source Seed Type Example Paths
github/rest-api-description Large OpenAPI 3.0 and 3.1 specs descriptions-next/api.github.com/api.github.com.yaml, descriptions-next/api.github.com/dereferenced/api.github.com.deref.yaml, descriptions-next/ghec/ghec.yaml
APIs-guru/openapi-directory Broad OpenAPI and Swagger corpus APIs/1password.com/events/1.2.0/openapi.yaml, APIs/ably.io/platform/1.1.0/openapi.yaml, APIs/adyen.com/CheckoutService/70/openapi.yaml, APIs/adafruit.com/2.0.0/swagger.yaml
OAI/OpenAPI-Specification Canonical OpenAPI examples _archive_/schemas/v3.0/pass/petstore.yaml, _archive_/schemas/v3.0/pass/petstore-expanded.yaml, _archive_/schemas/v3.0/pass/uspto.yaml, _archive_/schemas/v3.0/schema.yaml
argoproj/argo-cd Kubernetes, GitOps, ApplicationSet, Helm applicationset/examples/design-doc/applicationset.yaml, applicationset/examples/git-generator-directory/cluster-addons/prometheus-operator/values.yaml
prometheus/prometheus Good and intentionally bad app config config/testdata/conf.good.yml, cmd/promtool/testdata/prometheus-config.good.yml, cmd/promtool/testdata/prometheus-config.bad.yml, cmd/promtool/testdata/rules_large.yml
bitnami/charts Helm and Kubernetes chart YAML bitnami/airflow/values.yaml, bitnami/apache/Chart.yaml, bitnami/airflow/templates/web/deployment.yaml

The corpus should include API and non-API YAML:

  • OpenAPI 3.0, OpenAPI 3.1, and Swagger 2.0.
  • Kubernetes manifests.
  • Helm chart metadata, values files, and templates that are valid YAML.
  • GitHub Actions workflows.
  • Docker Compose files.
  • Prometheus configs and rule files.
  • Static-site, package, and application settings.
  • Intentionally bad real fixtures from upstream projects when available.

Manifest

Every seed must be listed in a manifest. The manifest should include:

  • stable seed ID,
  • repository owner and name,
  • commit SHA or release tag,
  • source path,
  • raw URL,
  • category,
  • byte size,
  • line count,
  • license or provenance note,
  • expected parser outcome,
  • expected validation outcome,
  • whether the seed belongs in the fast/default corpus,
  • whether the seed belongs only in the full opt-in corpus.

Huge files or files with awkward licensing should be downloaded by a corpus script rather than committed directly. Reduced fixtures that capture bugs should be committed in Tests/Fixtures/.

Corpus Sizes

Fast/default gate:

  • at least 25 pinned public real-world YAML seeds,
  • small enough for normal verification,
  • runs on macOS and Linux by default,
  • documents Windows and WASI depth if they run a smaller subset.

Full opt-in gate:

  • at least 100 pinned public real-world YAML seeds,
  • includes at least 20 very large OpenAPI files,
  • includes at least 20 non-API config files,
  • includes valid and intentionally invalid upstream fixtures where available.

Mutation Coverage

Mutations should start from real seeds and cover common production mistakes:

  • spacing after mapping colons,
  • indentation drift,
  • broken sequence markers,
  • tab indentation in sensitive positions,
  • malformed anchors and aliases,
  • unknown aliases,
  • broken quoted scalars,
  • broken literal and folded block scalars,
  • flow sequence and flow mapping mistakes,
  • duplicate mapping keys,
  • explicit key and explicit value mistakes,
  • stream boundary mistakes.

Generated malformed input must assert structured validation/report behavior, not only that parsing throws.

Properties

The generated gate should include deterministic properties such as:

  • successful parse and dump and parse stability where PureYAML owns both parse and dump semantics,
  • no crash on malformed input,
  • parser failures produce structured diagnostics when requested,
  • Yams differential agreement for inputs inside the shared supported subset,
  • every failure records the real seed ID and generator seed.

Every generated failure must be reproducible from a seed. Any non-trivial failure should be reduced into a stable checked-in regression fixture.

Artifacts

Corpus runs should write artifacts under .build/pureyaml-artifacts/.

.build/pureyaml-artifacts/
  real-seed-manifest.json
  real-seed-summary.yaml
  generated-validation-summary.json
  generated-validation-summary.yaml
  seeds.json
  failures/
    seed-id-000123.input.yaml
    seed-id-000123.mutated.yaml
    seed-id-000123.diagnostics.json
    seed-id-000123.reduced.yaml
  differential/
    yams-comparison.json
  environment.txt

CI should upload these artifacts with the current Node 24-compatible artifact action:

- name: Upload corpus artifacts
  if: always()
  uses: actions/upload-artifact@v6
  with:
    name: pureyaml-corpus-${{ runner.os }}-${{ github.sha }}
    path: .build/pureyaml-artifacts
    retention-days: 30

Artifacts are temporary. Issues and commits must link the relevant artifact while it exists, then preserve the important case as a committed fixture.

Acceptance Checklist

  • The fast corpus has at least 25 pinned public real-world YAML seeds.
  • The full corpus has at least 100 pinned public real-world YAML seeds.
  • The manifest records provenance and expected outcomes for every seed.
  • The mutation gate is derived from real seeds.
  • Generated failures are reproducible by seed.
  • Structured validation artifacts are produced even on failure.
  • Differential disagreements against Yams are fixed or documented with reduced reproducers.
  • New parser bugs are reduced into permanent fixtures or exact regression tests.