docs(corpus): document real seed gates
This commit is contained in:
parent
5f420cdf02
commit
9f68671c16
@ -7,7 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
No changes yet.
|
||||
### Added
|
||||
|
||||
- Document the real-world YAML corpus, fuzz/property gate, and CI artifact
|
||||
contract for production hardening.
|
||||
|
||||
## [0.1.1] - 2026-06-06
|
||||
|
||||
|
||||
@ -56,12 +56,14 @@ flowchart TB
|
||||
I49["#49 Structured validation reports"]:::todo
|
||||
I50["#50 False-positive diagnostic triage"]:::todo
|
||||
I51["#51 Bad-YAML mutation suite"]:::todo
|
||||
I55["#55 Real corpus and fuzz gates"]:::todo
|
||||
I52["#52 OpenAPIDoctor integration dry run"]:::todo
|
||||
I53["#53 Yams differential regression gate"]:::todo
|
||||
I48["#48 Parser throughput toward libyaml parity"]:::todo
|
||||
E54 --> I49
|
||||
E54 --> I50
|
||||
E54 --> I51
|
||||
E54 --> I55
|
||||
E54 --> I52
|
||||
E54 --> I53
|
||||
E54 --> I48
|
||||
@ -105,6 +107,8 @@ but it does not copy their implementation into `Sources/`. See
|
||||
|
||||
- [Usage](docs/USAGE.md): public APIs, validation, emitter options, and
|
||||
cross-platform gates.
|
||||
- [Corpus gates](docs/CORPUS.md): real-world YAML seed sources, generated
|
||||
mutations, artifact format, and hardening acceptance criteria.
|
||||
- [Migration and support boundaries](docs/MIGRATION.md): what PureYAML can
|
||||
replace today and which YAML features are deliberately unsupported.
|
||||
- [Release process](docs/RELEASE.md): verification commands, tag commands, and
|
||||
|
||||
176
docs/CORPUS.md
Normal file
176
docs/CORPUS.md
Normal file
@ -0,0 +1,176 @@
|
||||
# Corpus Gates
|
||||
|
||||
PureYAML's production hardening depends on real YAML first and generated cases
|
||||
second. Synthetic generators are useful, but they must be anchored in documents
|
||||
people actually maintain.
|
||||
|
||||
## Current State
|
||||
|
||||
The corpus gate is designed but not implemented yet. Implementation is tracked
|
||||
by:
|
||||
|
||||
- #54, the production hardening epic.
|
||||
- #55, the real-world seed, fuzz, and property-based validation gate.
|
||||
- #53, Yams differential testing.
|
||||
- #51, mutation-based bad-YAML validation fixtures.
|
||||
- #44, humongous real-world YAML specs.
|
||||
|
||||
## Goal
|
||||
|
||||
Build deterministic parser and validation gates that combine:
|
||||
|
||||
- pinned real-world YAML seeds,
|
||||
- original-file parser and validator checks,
|
||||
- deterministic valid and invalid mutations from those seeds,
|
||||
- property checks for parser and dumper stability,
|
||||
- differential comparison against Yams where the input is inside the shared
|
||||
supported subset,
|
||||
- structured artifacts that explain every failure.
|
||||
|
||||
Every discovered bug must become a reduced fixture or exact regression test.
|
||||
Artifacts are useful for triage, but committed tests are the permanent truth.
|
||||
|
||||
## Seed Sources
|
||||
|
||||
The first corpus should be built from public repositories with pinned commits.
|
||||
|
||||
| Source | Seed Type | Example Paths |
|
||||
|---|---|---|
|
||||
| `github/rest-api-description` | Large OpenAPI 3.0 and 3.1 specs | `descriptions-next/api.github.com/api.github.com.yaml`, `descriptions-next/api.github.com/dereferenced/api.github.com.deref.yaml`, `descriptions-next/ghec/ghec.yaml` |
|
||||
| `APIs-guru/openapi-directory` | Broad OpenAPI and Swagger corpus | `APIs/1password.com/events/1.2.0/openapi.yaml`, `APIs/ably.io/platform/1.1.0/openapi.yaml`, `APIs/adyen.com/CheckoutService/70/openapi.yaml`, `APIs/adafruit.com/2.0.0/swagger.yaml` |
|
||||
| `OAI/OpenAPI-Specification` | Canonical OpenAPI examples | `_archive_/schemas/v3.0/pass/petstore.yaml`, `_archive_/schemas/v3.0/pass/petstore-expanded.yaml`, `_archive_/schemas/v3.0/pass/uspto.yaml`, `_archive_/schemas/v3.0/schema.yaml` |
|
||||
| `argoproj/argo-cd` | Kubernetes, GitOps, ApplicationSet, Helm | `applicationset/examples/design-doc/applicationset.yaml`, `applicationset/examples/git-generator-directory/cluster-addons/prometheus-operator/values.yaml` |
|
||||
| `prometheus/prometheus` | Good and intentionally bad app config | `config/testdata/conf.good.yml`, `cmd/promtool/testdata/prometheus-config.good.yml`, `cmd/promtool/testdata/prometheus-config.bad.yml`, `cmd/promtool/testdata/rules_large.yml` |
|
||||
| `bitnami/charts` | Helm and Kubernetes chart YAML | `bitnami/airflow/values.yaml`, `bitnami/apache/Chart.yaml`, `bitnami/airflow/templates/web/deployment.yaml` |
|
||||
|
||||
The corpus should include API and non-API YAML:
|
||||
|
||||
- OpenAPI 3.0, OpenAPI 3.1, and Swagger 2.0.
|
||||
- Kubernetes manifests.
|
||||
- Helm chart metadata, values files, and templates that are valid YAML.
|
||||
- GitHub Actions workflows.
|
||||
- Docker Compose files.
|
||||
- Prometheus configs and rule files.
|
||||
- Static-site, package, and application settings.
|
||||
- Intentionally bad real fixtures from upstream projects when available.
|
||||
|
||||
## Manifest
|
||||
|
||||
Every seed must be listed in a manifest. The manifest should include:
|
||||
|
||||
- stable seed ID,
|
||||
- repository owner and name,
|
||||
- commit SHA or release tag,
|
||||
- source path,
|
||||
- raw URL,
|
||||
- category,
|
||||
- byte size,
|
||||
- line count,
|
||||
- license or provenance note,
|
||||
- expected parser outcome,
|
||||
- expected validation outcome,
|
||||
- whether the seed belongs in the fast/default corpus,
|
||||
- whether the seed belongs only in the full opt-in corpus.
|
||||
|
||||
Huge files or files with awkward licensing should be downloaded by a corpus
|
||||
script rather than committed directly. Reduced fixtures that capture bugs should
|
||||
be committed in `Tests/Fixtures/`.
|
||||
|
||||
## Corpus Sizes
|
||||
|
||||
Fast/default gate:
|
||||
|
||||
- at least 25 pinned public real-world YAML seeds,
|
||||
- small enough for normal verification,
|
||||
- runs on macOS and Linux by default,
|
||||
- documents Windows and WASI depth if they run a smaller subset.
|
||||
|
||||
Full opt-in gate:
|
||||
|
||||
- at least 100 pinned public real-world YAML seeds,
|
||||
- includes at least 20 very large OpenAPI files,
|
||||
- includes at least 20 non-API config files,
|
||||
- includes valid and intentionally invalid upstream fixtures where available.
|
||||
|
||||
## Mutation Coverage
|
||||
|
||||
Mutations should start from real seeds and cover common production mistakes:
|
||||
|
||||
- spacing after mapping colons,
|
||||
- indentation drift,
|
||||
- broken sequence markers,
|
||||
- tab indentation in sensitive positions,
|
||||
- malformed anchors and aliases,
|
||||
- unknown aliases,
|
||||
- broken quoted scalars,
|
||||
- broken literal and folded block scalars,
|
||||
- flow sequence and flow mapping mistakes,
|
||||
- duplicate mapping keys,
|
||||
- explicit key and explicit value mistakes,
|
||||
- stream boundary mistakes.
|
||||
|
||||
Generated malformed input must assert structured validation/report behavior, not
|
||||
only that parsing throws.
|
||||
|
||||
## Properties
|
||||
|
||||
The generated gate should include deterministic properties such as:
|
||||
|
||||
- successful parse and dump and parse stability where PureYAML owns both parse
|
||||
and dump semantics,
|
||||
- no crash on malformed input,
|
||||
- parser failures produce structured diagnostics when requested,
|
||||
- Yams differential agreement for inputs inside the shared supported subset,
|
||||
- every failure records the real seed ID and generator seed.
|
||||
|
||||
Every generated failure must be reproducible from a seed. Any non-trivial
|
||||
failure should be reduced into a stable checked-in regression fixture.
|
||||
|
||||
## Artifacts
|
||||
|
||||
Corpus runs should write artifacts under `.build/pureyaml-artifacts/`.
|
||||
|
||||
```text
|
||||
.build/pureyaml-artifacts/
|
||||
real-seed-manifest.json
|
||||
real-seed-summary.yaml
|
||||
generated-validation-summary.json
|
||||
generated-validation-summary.yaml
|
||||
seeds.json
|
||||
failures/
|
||||
seed-id-000123.input.yaml
|
||||
seed-id-000123.mutated.yaml
|
||||
seed-id-000123.diagnostics.json
|
||||
seed-id-000123.reduced.yaml
|
||||
differential/
|
||||
yams-comparison.json
|
||||
environment.txt
|
||||
```
|
||||
|
||||
CI should upload these artifacts with the current Node 24-compatible artifact
|
||||
action:
|
||||
|
||||
```yaml
|
||||
- name: Upload corpus artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: pureyaml-corpus-${{ runner.os }}-${{ github.sha }}
|
||||
path: .build/pureyaml-artifacts
|
||||
retention-days: 30
|
||||
```
|
||||
|
||||
Artifacts are temporary. Issues and commits must link the relevant artifact
|
||||
while it exists, then preserve the important case as a committed fixture.
|
||||
|
||||
## Acceptance Checklist
|
||||
|
||||
- The fast corpus has at least 25 pinned public real-world YAML seeds.
|
||||
- The full corpus has at least 100 pinned public real-world YAML seeds.
|
||||
- The manifest records provenance and expected outcomes for every seed.
|
||||
- The mutation gate is derived from real seeds.
|
||||
- Generated failures are reproducible by seed.
|
||||
- Structured validation artifacts are produced even on failure.
|
||||
- Differential disagreements against Yams are fixed or documented with reduced
|
||||
reproducers.
|
||||
- New parser bugs are reduced into permanent fixtures or exact regression tests.
|
||||
Loading…
Reference in New Issue
Block a user