From 9f68671c165cfa0552a614e8da60632218ebd1d1 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Sat, 6 Jun 2026 01:07:45 +0200 Subject: [PATCH] docs(corpus): document real seed gates --- CHANGELOG.md | 5 +- README.md | 4 ++ docs/CORPUS.md | 176 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 docs/CORPUS.md diff --git a/CHANGELOG.md b/CHANGELOG.md index afc0d14..c2b323f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -No changes yet. +### Added + +- Document the real-world YAML corpus, fuzz/property gate, and CI artifact + contract for production hardening. ## [0.1.1] - 2026-06-06 diff --git a/README.md b/README.md index 6249833..a1b8b97 100644 --- a/README.md +++ b/README.md @@ -56,12 +56,14 @@ flowchart TB I49["#49 Structured validation reports"]:::todo I50["#50 False-positive diagnostic triage"]:::todo I51["#51 Bad-YAML mutation suite"]:::todo + I55["#55 Real corpus and fuzz gates"]:::todo I52["#52 OpenAPIDoctor integration dry run"]:::todo I53["#53 Yams differential regression gate"]:::todo I48["#48 Parser throughput toward libyaml parity"]:::todo E54 --> I49 E54 --> I50 E54 --> I51 + E54 --> I55 E54 --> I52 E54 --> I53 E54 --> I48 @@ -105,6 +107,8 @@ but it does not copy their implementation into `Sources/`. See - [Usage](docs/USAGE.md): public APIs, validation, emitter options, and cross-platform gates. +- [Corpus gates](docs/CORPUS.md): real-world YAML seed sources, generated + mutations, artifact format, and hardening acceptance criteria. - [Migration and support boundaries](docs/MIGRATION.md): what PureYAML can replace today and which YAML features are deliberately unsupported. - [Release process](docs/RELEASE.md): verification commands, tag commands, and diff --git a/docs/CORPUS.md b/docs/CORPUS.md new file mode 100644 index 0000000..09596fd --- /dev/null +++ b/docs/CORPUS.md @@ -0,0 +1,176 @@ +# Corpus Gates + +PureYAML's production hardening depends on real YAML first and generated cases +second. Synthetic generators are useful, but they must be anchored in documents +people actually maintain. + +## Current State + +The corpus gate is designed but not implemented yet. Implementation is tracked +by: + +- #54, the production hardening epic. +- #55, the real-world seed, fuzz, and property-based validation gate. +- #53, Yams differential testing. +- #51, mutation-based bad-YAML validation fixtures. +- #44, humongous real-world YAML specs. + +## Goal + +Build deterministic parser and validation gates that combine: + +- pinned real-world YAML seeds, +- original-file parser and validator checks, +- deterministic valid and invalid mutations from those seeds, +- property checks for parser and dumper stability, +- differential comparison against Yams where the input is inside the shared + supported subset, +- structured artifacts that explain every failure. + +Every discovered bug must become a reduced fixture or exact regression test. +Artifacts are useful for triage, but committed tests are the permanent truth. + +## Seed Sources + +The first corpus should be built from public repositories with pinned commits. + +| Source | Seed Type | Example Paths | +|---|---|---| +| `github/rest-api-description` | Large OpenAPI 3.0 and 3.1 specs | `descriptions-next/api.github.com/api.github.com.yaml`, `descriptions-next/api.github.com/dereferenced/api.github.com.deref.yaml`, `descriptions-next/ghec/ghec.yaml` | +| `APIs-guru/openapi-directory` | Broad OpenAPI and Swagger corpus | `APIs/1password.com/events/1.2.0/openapi.yaml`, `APIs/ably.io/platform/1.1.0/openapi.yaml`, `APIs/adyen.com/CheckoutService/70/openapi.yaml`, `APIs/adafruit.com/2.0.0/swagger.yaml` | +| `OAI/OpenAPI-Specification` | Canonical OpenAPI examples | `_archive_/schemas/v3.0/pass/petstore.yaml`, `_archive_/schemas/v3.0/pass/petstore-expanded.yaml`, `_archive_/schemas/v3.0/pass/uspto.yaml`, `_archive_/schemas/v3.0/schema.yaml` | +| `argoproj/argo-cd` | Kubernetes, GitOps, ApplicationSet, Helm | `applicationset/examples/design-doc/applicationset.yaml`, `applicationset/examples/git-generator-directory/cluster-addons/prometheus-operator/values.yaml` | +| `prometheus/prometheus` | Good and intentionally bad app config | `config/testdata/conf.good.yml`, `cmd/promtool/testdata/prometheus-config.good.yml`, `cmd/promtool/testdata/prometheus-config.bad.yml`, `cmd/promtool/testdata/rules_large.yml` | +| `bitnami/charts` | Helm and Kubernetes chart YAML | `bitnami/airflow/values.yaml`, `bitnami/apache/Chart.yaml`, `bitnami/airflow/templates/web/deployment.yaml` | + +The corpus should include API and non-API YAML: + +- OpenAPI 3.0, OpenAPI 3.1, and Swagger 2.0. +- Kubernetes manifests. +- Helm chart metadata, values files, and templates that are valid YAML. +- GitHub Actions workflows. +- Docker Compose files. +- Prometheus configs and rule files. +- Static-site, package, and application settings. +- Intentionally bad real fixtures from upstream projects when available. + +## Manifest + +Every seed must be listed in a manifest. The manifest should include: + +- stable seed ID, +- repository owner and name, +- commit SHA or release tag, +- source path, +- raw URL, +- category, +- byte size, +- line count, +- license or provenance note, +- expected parser outcome, +- expected validation outcome, +- whether the seed belongs in the fast/default corpus, +- whether the seed belongs only in the full opt-in corpus. + +Huge files or files with awkward licensing should be downloaded by a corpus +script rather than committed directly. Reduced fixtures that capture bugs should +be committed in `Tests/Fixtures/`. + +## Corpus Sizes + +Fast/default gate: + +- at least 25 pinned public real-world YAML seeds, +- small enough for normal verification, +- runs on macOS and Linux by default, +- documents Windows and WASI depth if they run a smaller subset. + +Full opt-in gate: + +- at least 100 pinned public real-world YAML seeds, +- includes at least 20 very large OpenAPI files, +- includes at least 20 non-API config files, +- includes valid and intentionally invalid upstream fixtures where available. + +## Mutation Coverage + +Mutations should start from real seeds and cover common production mistakes: + +- spacing after mapping colons, +- indentation drift, +- broken sequence markers, +- tab indentation in sensitive positions, +- malformed anchors and aliases, +- unknown aliases, +- broken quoted scalars, +- broken literal and folded block scalars, +- flow sequence and flow mapping mistakes, +- duplicate mapping keys, +- explicit key and explicit value mistakes, +- stream boundary mistakes. + +Generated malformed input must assert structured validation/report behavior, not +only that parsing throws. + +## Properties + +The generated gate should include deterministic properties such as: + +- successful parse and dump and parse stability where PureYAML owns both parse + and dump semantics, +- no crash on malformed input, +- parser failures produce structured diagnostics when requested, +- Yams differential agreement for inputs inside the shared supported subset, +- every failure records the real seed ID and generator seed. + +Every generated failure must be reproducible from a seed. Any non-trivial +failure should be reduced into a stable checked-in regression fixture. + +## Artifacts + +Corpus runs should write artifacts under `.build/pureyaml-artifacts/`. + +```text +.build/pureyaml-artifacts/ + real-seed-manifest.json + real-seed-summary.yaml + generated-validation-summary.json + generated-validation-summary.yaml + seeds.json + failures/ + seed-id-000123.input.yaml + seed-id-000123.mutated.yaml + seed-id-000123.diagnostics.json + seed-id-000123.reduced.yaml + differential/ + yams-comparison.json + environment.txt +``` + +CI should upload these artifacts with the current Node 24-compatible artifact +action: + +```yaml +- name: Upload corpus artifacts + if: always() + uses: actions/upload-artifact@v6 + with: + name: pureyaml-corpus-${{ runner.os }}-${{ github.sha }} + path: .build/pureyaml-artifacts + retention-days: 30 +``` + +Artifacts are temporary. Issues and commits must link the relevant artifact +while it exists, then preserve the important case as a committed fixture. + +## Acceptance Checklist + +- The fast corpus has at least 25 pinned public real-world YAML seeds. +- The full corpus has at least 100 pinned public real-world YAML seeds. +- The manifest records provenance and expected outcomes for every seed. +- The mutation gate is derived from real seeds. +- Generated failures are reproducible by seed. +- Structured validation artifacts are produced even on failure. +- Differential disagreements against Yams are fixed or documented with reduced + reproducers. +- New parser bugs are reduced into permanent fixtures or exact regression tests.