NodetteHQ/PureYAML
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(parser): harden diagnostics and streaming

Browse Source 
Add structured validation diagnostics, JSON and Yams comparison gates, and lazy token/event parsing so production behavior is measured without retaining full intermediate parser arrays.
This commit is contained in:
Mihaela Mihaljevic 2026-06-06 07:57:48 +02:00
parent 9f68671c16
commit 208b892019
44 changed files with 3654 additions and 213 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
.github/workflows/ci.yml vendored
View File

@ -6,6 +6,9 @@ on:
push:
branches: [main]
pull_request:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1"
permissions:
contents: read
@ -86,3 +89,26 @@ jobs:
run: |
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh"
bash scripts/check-wasm.sh
corpus:
name: Corpus
if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
runs-on: macos-latest
steps:
- uses: actions/checkout@v6
- name: Run corpus gates
run: |
bash scripts/check-json-compatibility.sh
bash scripts/check-corpus.sh
bash scripts/check-yams-differential.sh
bash scripts/check-yams-diagnostics.sh
bash scripts/check-yams-validation.sh
bash scripts/check-throughput.sh
bash scripts/check-performance-phases.sh
- name: Upload corpus artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: pureyaml-corpus-${{ runner.os }}-${{ github.sha }}
path: .build/pureyaml-artifacts
retention-days: 30

45
CHANGELOG.md
View File

@ -7,10 +7,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.1.2] - 2026-06-06
### Added
- Document the real-world YAML corpus, fuzz/property gate, and CI artifact
contract for production hardening.
- Add machine-readable parser diagnostic codes to validation report payloads and
cover malformed production-shaped YAML with 10+ structured diagnostics.
- Vendor a 114-file real-world YAML corpus with a machine-readable manifest,
default metadata checks, and an opt-in full parse/validate/round-trip gate.
- Add large public APIs.guru Stripe and Zoom OpenAPI YAML specs to the opt-in
corpus gate.
- Add a reduced synthetic ACME Zephyr fixture set for Stitcher-style multi-file
OpenAPI inputs and bundled output.
- Add a full renamed Birch-style ACME fixture catalog with 594 YAML files,
preserving module/file structure while replacing private organization names,
domains, and documentation URLs.
- Add YAML 1.2 JSON-subset compatibility for compact flow mappings such as
`{"openapi":"3.0.0"}`, including regression coverage for colons inside flow
mapping values.
- Add JSON compatibility tests and artifacts that compare PureYAML against
Swift's JSON decoder across JSON roots, escapes, Unicode surrogate pairs,
exponent numbers, nested values, and whitespace.
- Preserve round-trip fidelity for mapping keys that require quotes, including
trailing-space keys found in the large Zoom OpenAPI corpus seed.
- Add an opt-in Yams differential script that compares parse behavior for the
checked-in real corpus without adding Yams to the PureYAML package manifest.
- Add an opt-in Yams diagnostic comparison script that records PureYAML
structured malformed-input diagnostics alongside Yams parser errors.
- Add an opt-in Yams validation comparison script proving structured duplicate
key validation on YAML that both parsers can load.
- Add an opt-in Yams throughput script and scheduled/manual CI artifact for
release-mode parser performance comparison on representative corpus seeds.
- Add an opt-in PureYAML phase profiler that times scanner, event parser,
composer, lazy parse, validation, and dumping phases on slow corpus seeds.
- Compose normal parser output from events as they are emitted, avoiding a full
intermediate event array while keeping `parseEvents` available for explicit
event materialization.
- Lazily scan tokens for normal `parse` and `parseStream` calls, avoiding a
retained full token array on the main parser path.
- Track reader positions with a UTF-8 cursor so ASCII-heavy scanner advancement
avoids per-character string allocation while preserving byte-accurate marks.
- Add deterministic generated corpus properties for valid YAML round trips and
malformed real-seed mutations with structured diagnostics.
- Write corpus run artifacts under `.build/pureyaml-artifacts/`, including real
seed summaries, generated mutation summaries, seed metadata, and environment
details.
- Add a scheduled/manual CI corpus job that runs the corpus and Yams
differential gates and uploads the generated artifact bundle.
## [0.1.1] - 2026-06-06

45
README.md
View File

@ -35,16 +35,19 @@ flowchart TB
LReview["PR in review"]:::review
LEpic["Epic grouping"]:::epic
LTodo["Open issue, no PR"]:::todo
LDone ~~~ LReview
LReview ~~~ LEpic
LEpic ~~~ LTodo
LDone --> LReview
LReview --> LEpic
LEpic --> LTodo
```
Parser Replacement Roadmap #8 is complete in main.
Production Hardening and Validation Readiness #54 is active. It tracks the
remaining production-readiness work after current Yams parse-success parity on
the checked corpus:
Production Hardening and Validation Readiness #54 is active. PureYAML now has
checked-corpus Yams parse-success parity, structured diagnostics and validation,
JSON compatibility proof gates, internal token/event streaming, real-corpus
property tests, and repeatable performance comparison artifacts. Remaining work
is concentrated in downstream integration, true chunked input streaming, broader
malformed fixtures, and continued throughput optimization:
```mermaid
flowchart TB
@ -53,25 +56,33 @@ flowchart TB
classDef epic fill:#f2e5ff,stroke:#af52de,color:#111827
classDef todo fill:#f2f4f7,stroke:#8e8e93,color:#111827
E54["#54 Production hardening epic"]:::epic
I49["#49 Structured validation reports"]:::todo
I50["#50 False-positive diagnostic triage"]:::todo
I49["#49 Structured validation reports"]:::done
I50["#50 False-positive diagnostic triage"]:::done
I51["#51 Bad-YAML mutation suite"]:::todo
I55["#55 Real corpus and fuzz gates"]:::todo
I55["#55 Corpus fuzz and property gates"]:::done
I44["#44 Humongous YAML corpus gate"]:::done
I45["#45 Yams execution and diagnostics comparison"]:::done
I47["#47 Yams-parseable OpenAPI gaps"]:::done
I52["#52 OpenAPIDoctor integration dry run"]:::todo
I53["#53 Yams differential regression gate"]:::todo
I53["#53 Yams differential regression gate"]:::done
I43["#43 True chunked input streaming"]:::todo
I48["#48 Parser throughput toward libyaml parity"]:::todo
E54 --> I49
E54 --> I50
E54 --> I51
E54 --> I55
E54 --> I52
E54 --> I53
E54 --> I48
I49 --> I50
I50 --> I51
I51 --> I55
I55 --> I44
I44 --> I45
I45 --> I47
I47 --> I53
I53 --> I52
I52 --> I43
I43 --> I48
```
## Status
The current release is 0.1.1. It includes block mappings, block sequences,
The current release is 0.1.2. It includes block mappings, block sequences,
ordered mappings, common scalars, quoted strings, comments, flow collections,
literal and folded block scalars, anchors, aliases, YAML directives, document
markers, explicit built-in scalar tags, merge-key expansion, complex mapping keys,

5
Sources/Emitting/Dumper.swift
View File

@ -352,10 +352,7 @@ extension PureYAML.Emitting.Dumper {
}
func escapeKey(_ key: String) -> String {
if key.isEmpty || key.contains(":") || key.contains("#") || key.contains("\n") {
return quote(key)
}
return key
canRenderPlainString(key) ? key : quote(key)
}
func quote(_ value: String) -> String {

318
Sources/Parsing/EventStreamComposer.swift Normal file
View File

@ -0,0 +1,318 @@
extension PureYAML.Parsing {
enum EventStreamComposerMode {
case singleDocument
case stream
}
struct EventStreamComposedNode {
var value: PureYAML.Model.Value
var mappingKey: PureYAML.Parsing.EventComposer.ComposedMappingKey
var mark: PureYAML.Parsing.Mark
var sequenceItems: [EventStreamComposedNode] = []
}
enum EventStreamContainerKind {
case sequence
case mapping
}
struct EventStreamContainer {
var kind: EventStreamContainerKind
var anchor: String?
var values: [EventStreamComposedNode] = []
var mergeSources: [[PureYAML.Model.Pair]] = []
var localPairs: [PureYAML.Model.Pair] = []
var pendingKey: PureYAML.Parsing.EventComposer.ComposedMappingKey?
var mark: PureYAML.Parsing.Mark
}
struct EventStreamComposer {
let scalarParser: Parser
let mode: EventStreamComposerMode
var anchors: [String: PureYAML.Model.Value] = [:]
var containers: [EventStreamContainer] = []
var documentValues: [PureYAML.Model.Value] = []
var currentDocumentValue: PureYAML.Model.Value?
var didSeeStreamStart = false
var didSeeStreamEnd = false
init(
scalarParser: Parser,
mode: EventStreamComposerMode,
) {
self.scalarParser = scalarParser
self.mode = mode
}
mutating func consume(_ event: PureYAML.Parsing.Event) throws {
switch event {
case .streamStart:
didSeeStreamStart = true
case .streamEnd:
didSeeStreamEnd = true
case let .documentStart(mark):
try startDocument(mark: mark)
case let .documentEnd(mark):
try endDocument(mark: mark)
case let .scalar(value, anchor, tag, style, mark):
try consumeScalar(value, anchor: anchor, tag: tag, style: style, mark: mark)
case let .alias(anchor, mark):
try consumeAlias(anchor, mark: mark)
case let .sequenceStart(anchor, _, _, mark):
containers.append(.init(kind: .sequence, anchor: anchor, mark: mark))
case let .sequenceEnd(mark):
try completeSequence(mark: mark)
case let .mappingStart(anchor, _, _, mark):
containers.append(.init(kind: .mapping, anchor: anchor, mark: mark))
case let .mappingEnd(mark):
try completeMapping(mark: mark)
}
}
mutating func composedValue() throws -> PureYAML.Model.Value {
guard didSeeStreamStart, didSeeStreamEnd else {
throw unexpectedEvent(expected: "complete YAML stream", mark: .start)
}
guard documentValues.count == 1, let value = documentValues.first else {
throw unexpectedEvent(expected: "single YAML document", mark: .start)
}
return value
}
mutating func composedStream() throws -> [PureYAML.Stream.Document] {
guard didSeeStreamStart, didSeeStreamEnd else {
throw unexpectedEvent(expected: "complete YAML stream", mark: .start)
}
return documentValues.enumerated().map { index, value in
PureYAML.Stream.Document(index: index, value: value)
}
}
}
}
private extension PureYAML.Parsing.EventStreamComposer {
mutating func startDocument(mark: PureYAML.Parsing.Mark) throws {
guard containers.isEmpty else {
throw unexpectedEvent(expected: "document content end", mark: mark)
}
if mode == .singleDocument, !documentValues.isEmpty {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: mark.line)
}
anchors = [:]
currentDocumentValue = nil
}
mutating func endDocument(mark: PureYAML.Parsing.Mark) throws {
guard containers.isEmpty else {
throw unexpectedEvent(expected: "document content end", mark: mark)
}
documentValues.append(currentDocumentValue ?? .null)
currentDocumentValue = nil
}
mutating func consumeScalar(
_ value: String,
anchor: String?,
tag: String?,
style: PureYAML.Parsing.ScalarStyle,
mark: PureYAML.Parsing.Mark,
) throws {
let model = try scalarParser.composeScalarValue(value, tag: tag, style: style, mark: mark)
if let anchor {
anchors[anchor] = .string(value)
}
try complete(.init(
value: model,
mappingKey: .init(
keyNode: .string(value),
scalarValue: value,
tag: scalarParser.normalizedTag(tag),
style: style,
),
mark: mark,
))
}
mutating func consumeAlias(
_ anchor: String,
mark: PureYAML.Parsing.Mark,
) throws {
guard let value = anchors[anchor] else {
throw PureYAML.Parsing.ParseError.undefinedAlias(
anchor: anchor,
line: mark.line,
column: mark.column,
)
}
try complete(.init(
value: value,
mappingKey: .init(
keyNode: PureYAML.Model.Key(value: value),
scalarValue: nil,
tag: nil,
style: nil,
),
mark: mark,
))
}
mutating func completeSequence(mark: PureYAML.Parsing.Mark) throws {
guard let container = containers.popLast(), container.kind == .sequence else {
throw unexpectedEvent(expected: "sequence start", mark: mark)
}
let value = PureYAML.Model.Value.sequence(container.values.map(\.value))
store(value, anchor: container.anchor)
try complete(.init(
value: value,
mappingKey: .init(
keyNode: PureYAML.Model.Key(value: value),
scalarValue: nil,
tag: nil,
style: nil,
),
mark: container.mark,
sequenceItems: container.values,
))
}
mutating func completeMapping(mark: PureYAML.Parsing.Mark) throws {
guard let container = containers.popLast(), container.kind == .mapping else {
throw unexpectedEvent(expected: "mapping start", mark: mark)
}
guard container.pendingKey == nil else {
throw unexpectedEvent(expected: "mapping value", mark: mark)
}
let pairs = mergedPairs(
from: container.mergeSources,
localPairs: container.localPairs,
)
let value = PureYAML.Model.Value.mapping(.init(pairs))
store(value, anchor: container.anchor)
try complete(.init(
value: value,
mappingKey: .init(
keyNode: PureYAML.Model.Key(value: value),
scalarValue: nil,
tag: nil,
style: nil,
),
mark: container.mark,
))
}
mutating func complete(_ node: PureYAML.Parsing.EventStreamComposedNode) throws {
guard !containers.isEmpty else {
currentDocumentValue = node.value
return
}
switch containers[containers.count - 1].kind {
case .sequence:
containers[containers.count - 1].values.append(node)
case .mapping:
try completeMappingNode(node)
}
}
mutating func completeMappingNode(_ node: PureYAML.Parsing.EventStreamComposedNode) throws {
let index = containers.count - 1
guard let key = containers[index].pendingKey else {
containers[index].pendingKey = node.mappingKey
return
}
if key.isMergeKey {
try containers[index].mergeSources.append(contentsOf: mergeSources(
from: node,
))
} else {
containers[index].localPairs.append(.init(keyNode: key.keyNode, value: node.value))
}
containers[index].pendingKey = nil
}
mutating func store(
_ value: PureYAML.Model.Value,
anchor: String?,
) {
guard let anchor else {
return
}
anchors[anchor] = value
}
func mergeSources(
from node: PureYAML.Parsing.EventStreamComposedNode,
) throws -> [[PureYAML.Model.Pair]] {
switch node.value {
case let .mapping(mapping):
return [mapping.pairs]
case let .sequence(values):
guard !node.sequenceItems.isEmpty else {
return try values.map { value in
guard case let .mapping(mapping) = value else {
throw invalidMergeValue(mark: node.mark)
}
return mapping.pairs
}
}
return try values.enumerated().map { index, value in
let mark = node.sequenceItems[index].mark
guard case let .mapping(mapping) = value else {
throw invalidMergeValue(mark: mark)
}
return mapping.pairs
}
case .bool, .double, .int, .null, .string:
throw invalidMergeValue(mark: node.mark)
}
}
func mergedPairs(
from mergeSources: [[PureYAML.Model.Pair]],
localPairs: [PureYAML.Model.Pair],
) -> [PureYAML.Model.Pair] {
guard !mergeSources.isEmpty else {
return localPairs
}
let localKeys = Set(localPairs.map(\.keyNode))
var inheritedKeys = Set<PureYAML.Model.Key>()
var inheritedPairs: [PureYAML.Model.Pair] = []
for source in mergeSources {
var sourceKeys = Set<PureYAML.Model.Key>()
for pair in source {
guard !localKeys.contains(pair.keyNode),
!inheritedKeys.contains(pair.keyNode)
else {
continue
}
inheritedPairs.append(pair)
sourceKeys.insert(pair.keyNode)
}
inheritedKeys.formUnion(sourceKeys)
}
return inheritedPairs + localPairs
}
func invalidMergeValue(
mark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.invalidMergeValue(
line: mark.line,
column: mark.column,
)
}
func unexpectedEvent(
expected: String,
mark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.unexpectedEvent(
expected: expected,
actual: "streaming event",
line: mark.line,
column: mark.column,
)
}
}

96
Sources/Parsing/ParseError.swift
View File

@ -63,5 +63,101 @@ public extension PureYAML.Parsing {
"expected \(expected) at line \(line), column \(column), found \(actual)"
}
}
public var diagnosticCode: String {
switch self {
case .emptyDocument:
"emptyDocument"
case .tabIndentation:
"tabIndentation"
case .unexpectedIndentation:
"unexpectedIndentation"
case .mixedCollectionStyles:
"mixedCollectionStyles"
case .expectedMappingKey:
"expectedMappingKey"
case .expectedAnchorName:
"expectedAnchorName"
case .expectedAliasName:
"expectedAliasName"
case .expectedNode:
"expectedNode"
case .expectedScalarKey:
"expectedScalarKey"
case .incompatibleYAMLDirective:
"incompatibleYAMLDirective"
case .invalidTaggedScalar:
"invalidTaggedScalar"
case .invalidMergeValue:
"invalidMergeValue"
case .unterminatedTag:
"unterminatedTag"
case .unterminatedQuotedString:
"unterminatedQuotedString"
case .unsupportedDirective:
"unsupportedDirective"
case .unsupportedMultiDocumentStream:
"unsupportedMultiDocumentStream"
case .undefinedAlias:
"undefinedAlias"
case .unexpectedEvent:
"unexpectedEvent"
case .unexpectedToken:
"unexpectedToken"
}
}
public var sourceLine: Int? {
switch self {
case .emptyDocument:
nil
case let .tabIndentation(line),
let .unexpectedIndentation(line),
let .mixedCollectionStyles(line),
let .expectedMappingKey(line),
let .expectedAnchorName(line),
let .expectedAliasName(line),
let .incompatibleYAMLDirective(line),
let .unterminatedTag(line),
let .unterminatedQuotedString(line),
let .unsupportedDirective(_, line),
let .unsupportedMultiDocumentStream(line):
line
case let .expectedNode(line, _),
let .expectedScalarKey(line, _),
let .invalidTaggedScalar(_, _, line, _),
let .invalidMergeValue(line, _),
let .undefinedAlias(_, line, _),
let .unexpectedEvent(_, _, line, _),
let .unexpectedToken(_, _, line, _):
line
}
}
public var sourceColumn: Int? {
switch self {
case .emptyDocument,
.tabIndentation,
.unexpectedIndentation,
.mixedCollectionStyles,
.expectedMappingKey,
.expectedAnchorName,
.expectedAliasName,
.incompatibleYAMLDirective,
.unterminatedTag,
.unterminatedQuotedString,
.unsupportedDirective,
.unsupportedMultiDocumentStream:
nil
case let .expectedNode(_, column),
let .expectedScalarKey(_, column),
let .invalidTaggedScalar(_, _, _, column),
let .invalidMergeValue(_, column),
let .undefinedAlias(_, _, column),
let .unexpectedEvent(_, _, _, column),
let .unexpectedToken(_, _, _, column):
column
}
}
}
}

32
Sources/Parsing/Parser.Events.swift
View File

@ -4,4 +4,36 @@ extension PureYAML.Parsing.Parser {
var parser = PureYAML.Parsing.TokenEventParser(tokens: tokens, scalarParser: self)
return try parser.parse()
}
func parseStreaming(_ yaml: String) throws -> PureYAML.Model.Value {
var composer = PureYAML.Parsing.EventStreamComposer(
scalarParser: self,
mode: .singleDocument,
)
var parser = PureYAML.Parsing.TokenEventParser(
yaml: yaml,
scalarParser: self,
eventSink: { event in
try composer.consume(event)
},
)
try parser.parseStreaming()
return try composer.composedValue()
}
func parseStreamStreaming(_ yaml: String) throws -> [PureYAML.Stream.Document] {
var composer = PureYAML.Parsing.EventStreamComposer(
scalarParser: self,
mode: .stream,
)
var parser = PureYAML.Parsing.TokenEventParser(
yaml: yaml,
scalarParser: self,
eventSink: { event in
try composer.consume(event)
},
)
try parser.parseStreaming()
return try composer.composedStream()
}
}

8
Sources/Parsing/Parser.swift
View File

@ -4,15 +4,11 @@ public extension PureYAML.Parsing {
public init() {}
public func parse(_ yaml: String) throws -> PureYAML.Model.Value {
let events = try parseEvents(yaml)
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: self)
return try composer.compose()
try parseStreaming(yaml)
}
public func parseStream(_ yaml: String) throws -> [PureYAML.Stream.Document] {
let events = try parseEvents(yaml)
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: self)
return try composer.composeStream()
try parseStreamStreaming(yaml)
}
}
}

56
Sources/Parsing/Reader.swift
View File

@ -1,14 +1,18 @@
extension PureYAML.Parsing {
struct Reader {
private let source: String
private let utf8: String.UTF8View
private var position: String.Index
private var utf8Position: String.UTF8View.Index
private var line: Int
private var column: Int
private var utf8Index: Int
init(_ source: String) {
self.source = source
utf8 = source.utf8
position = source.startIndex
utf8Position = source.utf8.startIndex
line = 1
column = 1
utf8Index = 0
@ -42,22 +46,20 @@ extension PureYAML.Parsing {
return nil
}
let character = source[position]
position = source.index(after: position)
utf8Index += String(character).utf8.count
if character == "\r\n" {
switch utf8[utf8Position] {
case 13:
advanceCarriageReturn()
line += 1
column = 1
} else if character == "\r" {
if position < source.endIndex, source[position] == "\n" {
position = source.index(after: position)
utf8Index += 1
}
case 10:
advanceASCIICharacter()
line += 1
column = 1
} else if character == "\n" {
line += 1
column = 1
} else {
case 0 ..< 128:
advanceASCIICharacter()
column += 1
default:
advanceExtendedCharacter()
column += 1
}
return character
@ -71,5 +73,35 @@ extension PureYAML.Parsing {
}
return output
}
private mutating func advanceASCIICharacter() {
position = source.index(after: position)
utf8Position = utf8.index(after: utf8Position)
utf8Index += 1
}
private mutating func advanceCarriageReturn() {
position = source.index(after: position)
utf8Position = utf8.index(after: utf8Position)
utf8Index += 1
guard utf8Position < utf8.endIndex, utf8[utf8Position] == 10 else {
return
}
utf8Position = utf8.index(after: utf8Position)
utf8Index += 1
if position < source.endIndex, source[position] == "\n" {
position = source.index(after: position)
}
}
private mutating func advanceExtendedCharacter() {
position = source.index(after: position)
guard let nextUTF8Position = position.samePosition(in: utf8) else {
utf8Index += String(source[source.index(before: position)]).utf8.count
return
}
utf8Index += utf8.distance(from: utf8Position, to: nextUTF8Position)
utf8Position = nextUTF8Position
}
}
}

109
Sources/Parsing/Scalar.swift
View File

@ -137,35 +137,122 @@ extension PureYAML.Parsing.Parser {
}
let inner = text.dropFirst().dropLast()
var output = ""
var escaping = false
for character in inner {
if escaping {
appendDoubleQuotedEscape(character, to: &output)
escaping = false
} else if character == "\\" {
escaping = true
var index = inner.startIndex
while index < inner.endIndex {
let character = inner[index]
if character == "\\" {
inner.formIndex(after: &index)
guard index < inner.endIndex else {
output.append("\\")
break
}
appendDoubleQuotedEscape(in: inner, index: &index, to: &output)
} else {
output.append(character)
inner.formIndex(after: &index)
}
}
if escaping {
output.append("\\")
}
return output
}
func appendDoubleQuotedEscape(
_ character: Character,
in text: Substring,
index: inout Substring.Index,
to output: inout String,
) {
let character = text[index]
switch character {
case "\"": output.append("\"")
case "\\": output.append("\\")
case "/": output.append("/")
case "b": output.append("\u{08}")
case "f": output.append("\u{0C}")
case "n": output.append("\n")
case "r": output.append("\r")
case "t": output.append("\t")
case "u":
appendUnicodeEscape(in: text, index: &index, to: &output)
default: output.append(character)
}
text.formIndex(after: &index)
}
func appendUnicodeEscape(
in text: Substring,
index: inout Substring.Index,
to output: inout String,
) {
let escapeStart = index
guard let codeUnit = readUnicodeEscapeCodeUnit(in: text, index: &index) else {
output.append("u")
return
}
let surrogatePair = readSurrogatePairIfNeeded(high: codeUnit, in: text, index: &index)
if let scalar = surrogatePair {
output.unicodeScalars.append(scalar)
} else if let scalar = UnicodeScalar(codeUnit) {
output.unicodeScalars.append(scalar)
} else {
output += text[escapeStart ... index]
}
}
func readSurrogatePairIfNeeded(
high: UInt32,
in text: Substring,
index: inout Substring.Index,
) -> UnicodeScalar? {
guard isHighSurrogate(high) else {
return nil
}
return readSurrogatePair(high: high, in: text, index: &index)
}
func readSurrogatePair(
high: UInt32,
in text: Substring,
index: inout Substring.Index,
) -> UnicodeScalar? {
var probe = index
text.formIndex(after: &probe)
guard probe < text.endIndex, text[probe] == "\\" else {
return nil
}
text.formIndex(after: &probe)
guard probe < text.endIndex, text[probe] == "u",
let low = readUnicodeEscapeCodeUnit(in: text, index: &probe),
isLowSurrogate(low)
else {
return nil
}
index = probe
let value = 0x10000 + ((high - 0xD800) << 10) + (low - 0xDC00)
return UnicodeScalar(value)
}
func readUnicodeEscapeCodeUnit(
in text: Substring,
index: inout Substring.Index,
) -> UInt32? {
var probe = index
text.formIndex(after: &probe)
var hex = ""
for _ in 0 ..< 4 {
guard probe < text.endIndex, text[probe].isHexDigit else {
return nil
}
hex.append(text[probe])
text.formIndex(after: &probe)
}
index = text.index(before: probe)
return UInt32(hex, radix: 16)
}
func isHighSurrogate(_ value: UInt32) -> Bool {
(0xD800 ... 0xDBFF).contains(value)
}
func isLowSurrogate(_ value: UInt32) -> Bool {
(0xDC00 ... 0xDFFF).contains(value)
}
}

11
Sources/Parsing/Scanner.FlowContext.swift Normal file
View File

@ -0,0 +1,11 @@
extension PureYAML.Parsing {
enum ScannerFlowContextKind {
case sequence
case mapping
}
struct ScannerFlowContext {
var kind: ScannerFlowContextKind
var isExpectingKey: Bool
}
}

15
Sources/Parsing/Scanner.Helpers.swift
View File

@ -234,7 +234,8 @@ extension PureYAML.Parsing.Scanner {
func shouldEndPlainScalarAtMappingValue(state: State) -> Bool {
if state.flowDepth > 0 {
return isMappingValueBoundary(state.reader.peek(offset: 1))
return state.isExpectingFlowMappingKey
|| isMappingValueBoundary(state.reader.peek(offset: 1))
}
return isBlockMappingValueBoundary(state.reader.peek(offset: 1))
}
@ -242,4 +243,16 @@ extension PureYAML.Parsing.Scanner {
func isAtMappingValueIndicator(state: State) -> Bool {
state.reader.peek() == ":" && shouldEndPlainScalarAtMappingValue(state: state)
}
@discardableResult
func appendMappingKeyIfNeeded(
start: PureYAML.Parsing.Mark,
state: inout State,
) -> Bool {
guard isAtMappingValueIndicator(state: state) else {
return false
}
state.append(.mappingKey, mark: start, endMark: start)
return true
}
}

30
Sources/Parsing/Scanner.swift
View File

@ -47,6 +47,7 @@ extension PureYAML.Parsing.Scanner {
var hasDocumentStartMarker = false
var isDocumentClosed = false
var flowDepth = 0
var flowContexts: [PureYAML.Parsing.ScannerFlowContext] = []
var validImplicitIndentation: Set<Int> = []
var pendingBlockScalarIndentation = false
var pendingBlockScalarIndentationIndicator: Int?
@ -80,6 +81,11 @@ extension PureYAML.Parsing.Scanner {
hasDocumentContent = true
}
}
var isExpectingFlowMappingKey: Bool {
flowContexts.last?.kind == .mapping
&& flowContexts.last?.isExpectingKey == true
}
}
}
@ -123,7 +129,7 @@ extension PureYAML.Parsing.Scanner {
case "?" where isIndicatorBoundary(state.reader.peek(offset: 1)):
scanFixedToken(.mappingKey, state: &state)
return true
case ":" where isMappingValueBoundary(state.reader.peek(offset: 1)):
case ":" where isAtMappingValueIndicator(state: state):
scanFixedToken(.mappingValue, state: &state)
return true
default:
@ -138,22 +144,29 @@ extension PureYAML.Parsing.Scanner {
switch character {
case "[":
state.flowDepth += 1
state.flowContexts.append(.init(kind: .sequence, isExpectingKey: false))
scanFixedToken(.flowSequenceStart, state: &state)
return true
case "]":
state.flowDepth = max(0, state.flowDepth - 1)
scanFixedToken(.flowSequenceEnd, state: &state)
_ = state.flowContexts.popLast()
return true
case "{":
state.flowDepth += 1
state.flowContexts.append(.init(kind: .mapping, isExpectingKey: true))
scanFixedToken(.flowMappingStart, state: &state)
return true
case "}":
state.flowDepth = max(0, state.flowDepth - 1)
scanFixedToken(.flowMappingEnd, state: &state)
_ = state.flowContexts.popLast()
return true
case ",":
scanFixedToken(.flowEntry, state: &state)
if state.flowContexts.last?.kind == .mapping {
state.flowContexts[state.flowContexts.count - 1].isExpectingKey = true
}
return true
default:
return false
@ -225,6 +238,9 @@ extension PureYAML.Parsing.Scanner {
if case .blockEntry = kind, hasNodeOnSameLine(afterCurrentPositionIn: state.reader) {
state.validImplicitIndentation.insert(start.column + 1)
}
if case .mappingValue = kind, state.flowContexts.last?.kind == .mapping {
state.flowContexts[state.flowContexts.count - 1].isExpectingKey = false
}
}
func scanComment(_ state: inout State) {
@ -371,16 +387,4 @@ extension PureYAML.Parsing.Scanner {
}
state.append(.scalar(value: trimmed, style: .plain), mark: start, endMark: state.reader.mark)
}
@discardableResult
func appendMappingKeyIfNeeded(
start: PureYAML.Parsing.Mark,
state: inout State,
) -> Bool {
guard state.reader.peek() == ":", isMappingValueBoundary(state.reader.peek(offset: 1)) else {
return false
}
state.append(.mappingKey, mark: start, endMark: start)
return true
}
}

105
Sources/Parsing/TokenCursor.swift
View File

@ -1,11 +1,36 @@
extension PureYAML.Parsing {
private final class TokenScannerStateBox {
var state: Scanner.State
init(state: Scanner.State) {
self.state = state
}
}
struct TokenCursor {
private var tokens: [Token]
private(set) var index: Int
private var previousToken: Token?
private var scanner: Scanner?
private var scannerBox: TokenScannerStateBox?
init(tokens: [Token]) {
self.tokens = tokens.filter { !$0.kind.isComment }
index = 0
previousToken = nil
scanner = nil
scannerBox = nil
}
init(yaml: String, scanner: Scanner = .init()) {
var state = Scanner.State(reader: Reader(yaml))
state.append(.streamStart)
tokens = state.tokens.filter { !$0.kind.isComment }
state.tokens.removeAll(keepingCapacity: true)
index = 0
previousToken = nil
self.scanner = scanner
scannerBox = TokenScannerStateBox(state: state)
}
var current: Token? {
@ -16,14 +41,11 @@ extension PureYAML.Parsing {
}
var previous: Token? {
let previousIndex = index - 1
guard tokens.indices.contains(previousIndex) else {
return nil
}
return tokens[previousIndex]
previousToken
}
func peek(offset: Int = 1) -> Token? {
mutating func peek(offset: Int = 1) throws -> Token? {
try ensureAvailable(offset: offset)
let targetIndex = index + offset
guard tokens.indices.contains(targetIndex) else {
return nil
@ -31,13 +53,80 @@ extension PureYAML.Parsing {
return tokens[targetIndex]
}
mutating func advance() -> Token? {
guard let token = current else {
mutating func advance() throws -> Token? {
try ensureAvailable(offset: 0)
guard tokens.indices.contains(index) else {
return nil
}
let token = tokens[index]
previousToken = token
index += 1
compactConsumedTokens()
try ensureAvailable(offset: 0)
return token
}
private mutating func ensureAvailable(offset: Int) throws {
while tokens.count <= index + offset, scannerBox != nil {
try scanMoreTokens()
}
}
private mutating func scanMoreTokens() throws {
guard let scanner, let scannerBox else {
return
}
while scannerBox.state.tokens.isEmpty {
guard !scannerBox.state.reader.isAtEnd else {
scanner.closeIndentation(&scannerBox.state)
scannerBox.state.append(.streamEnd)
drainScannedTokens(from: &scannerBox.state)
self.scannerBox = nil
return
}
if scanner.isLineBreak(scannerBox.state.reader.peek()) {
scannerBox.state.reader.advance()
scannerBox.state.isAtLineStart = true
continue
}
if scannerBox.state.isAtLineStart {
try scanner.scanIndentation(&scannerBox.state)
if scannerBox.state.reader.isAtEnd {
continue
}
if scanner.isLineBreak(scannerBox.state.reader.peek()) {
continue
}
if try scanner.scanDirectiveOrDocumentMarker(&scannerBox.state) {
if !scannerBox.state.tokens.isEmpty {
break
}
continue
}
scannerBox.state.isAtLineStart = false
}
try scanner.scanToken(&scannerBox.state)
}
drainScannedTokens(from: &scannerBox.state)
}
private mutating func drainScannedTokens(from state: inout Scanner.State) {
tokens.append(contentsOf: state.tokens.filter { !$0.kind.isComment })
state.tokens.removeAll(keepingCapacity: true)
}
private mutating func compactConsumedTokens() {
guard index > 4096 else {
return
}
tokens.removeFirst(index)
index = 0
}
}
}

20
Sources/Parsing/TokenEventParser.Block.swift
View File

@ -6,7 +6,7 @@ extension PureYAML.Parsing.TokenEventParser {
let start = try expect("block sequence entry") { $0.isBlockEntry }
let startMark = properties.mark ?? start.mark
var endMark = start.endMark
events.append(.sequenceStart(
try emit(.sequenceStart(
anchor: properties.anchor,
tag: properties.tag,
style: .block,
@ -34,13 +34,13 @@ extension PureYAML.Parsing.TokenEventParser {
throw PureYAML.Parsing.ParseError.mixedCollectionStyles(line: current.mark.line)
}
events.append(.sequenceEnd(mark: endMark))
try emit(.sequenceEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .sequence, endMark: endMark)
}
mutating func consumeSequenceItemBoundaryDedent(sequenceIndentWidth: Int) throws {
guard dedentWidth(cursor.current) == sequenceIndentWidth,
cursor.peek()?.kind.isBlockEntry == true
try cursor.peek()?.kind.isBlockEntry == true
else {
return
}
@ -49,7 +49,7 @@ extension PureYAML.Parsing.TokenEventParser {
mutating func consumeSequenceItemBoundaryIndent(sequenceIndentWidth: Int) throws {
guard indentWidth(cursor.current) == sequenceIndentWidth,
cursor.peek()?.kind.isBlockEntry == true
try cursor.peek()?.kind.isBlockEntry == true
else {
return
}
@ -62,7 +62,7 @@ extension PureYAML.Parsing.TokenEventParser {
allowMappingKeyTerminator: Bool,
) throws {
if hasNodeOnSameLine(after: entry) {
let properties = consumeProperties()
let properties = try consumeProperties()
if cursor.current?.kind.isIndent == true {
_ = try expect("indent") { $0.isIndent }
endMark = try parseNode(properties: properties).endMark
@ -89,7 +89,7 @@ extension PureYAML.Parsing.TokenEventParser {
_ = try expect("dedent") { $0.isDedent }
} else {
let mark = entry.endMark
events.append(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
endMark = mark
}
}
@ -104,7 +104,7 @@ extension PureYAML.Parsing.TokenEventParser {
let contentColumn = min(startMark.column, tokenColumn)
let boundaryColumn = minimumColumn ?? contentColumn
var endMark = startMark
events.append(.mappingStart(
try emit(.mappingStart(
anchor: properties.anchor,
tag: properties.tag,
style: .block,
@ -124,7 +124,7 @@ extension PureYAML.Parsing.TokenEventParser {
column: current.mark.column,
)
}
events.append(.mappingEnd(mark: endMark))
try emit(.mappingEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: endMark)
}
@ -196,7 +196,7 @@ extension PureYAML.Parsing.TokenEventParser {
guard case let .scalar(value, style) = cursor.current?.kind else {
throw unexpectedToken(expected: "block scalar content")
}
let token = cursor.advance() ?? header
let token = try cursor.advance() ?? header
try lines.append(decodeScalar(value, style: style, line: token.mark.line))
endMark = token.endMark
}
@ -205,7 +205,7 @@ extension PureYAML.Parsing.TokenEventParser {
let value = blockScalarValue(lines: lines, header: header.kind)
let mark = properties.mark ?? header.mark
events.append(.scalar(
try emit(.scalar(
value: value,
anchor: properties.anchor,
tag: properties.tag,

16
Sources/Parsing/TokenEventParser.Flow.swift
View File

@ -10,7 +10,7 @@ extension PureYAML.Parsing.TokenEventParser {
}
let startMark = properties.mark ?? start.mark
var endMark = start.endMark
events.append(.sequenceStart(
try emit(.sequenceStart(
anchor: properties.anchor,
tag: properties.tag,
style: .flow,
@ -25,11 +25,11 @@ extension PureYAML.Parsing.TokenEventParser {
return false
}
endMark = end.endMark
events.append(.sequenceEnd(mark: endMark))
try emit(.sequenceEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .sequence, endMark: endMark)
}
if case .flowEntry = cursor.current?.kind {
_ = cursor.advance()
_ = try cursor.advance()
continue
}
endMark = try parseNode().endMark
@ -51,7 +51,7 @@ extension PureYAML.Parsing.TokenEventParser {
}
let startMark = properties.mark ?? start.mark
var endMark = start.endMark
events.append(.mappingStart(
try emit(.mappingStart(
anchor: properties.anchor,
tag: properties.tag,
style: .flow,
@ -66,11 +66,11 @@ extension PureYAML.Parsing.TokenEventParser {
return false
}
endMark = end.endMark
events.append(.mappingEnd(mark: endMark))
try emit(.mappingEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: endMark)
}
if case .flowEntry = cursor.current?.kind {
_ = cursor.advance()
_ = try cursor.advance()
continue
}
endMark = try parseFlowMappingPair().endMark
@ -94,10 +94,10 @@ extension PureYAML.Parsing.TokenEventParser {
return false
}
let properties = consumeProperties()
let properties = try consumeProperties()
guard properties != .none || cursor.current?.kind.isFlowTerminator == false else {
let mark = valueIndicator.endMark
events.append(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
}
return try parseNode(properties: properties)

40
Sources/Parsing/TokenEventParser.Helpers.swift
View File

@ -9,7 +9,7 @@ extension PureYAML.Parsing.TokenEventParser {
return false
}
let properties = consumeProperties()
let properties = try consumeProperties()
if cursor.current?.kind.isIndent == true {
_ = try expect("indent") { $0.isIndent }
let result: PureYAML.Parsing.NodeResult = if cursor.current?.kind.isBlockEntry == true {
@ -27,7 +27,7 @@ extension PureYAML.Parsing.TokenEventParser {
}
return result
}
if properties == .none, canParsePlainScalarWithContinuation(after: valueIndicator) {
if properties == .none, try canParsePlainScalarWithContinuation(after: valueIndicator) {
return try parsePlainScalarWithContinuation(properties: properties)
}
if cursor.current?.kind.isMappingKey == true, hasNodeOnSameLine(after: valueIndicator) {
@ -48,7 +48,7 @@ extension PureYAML.Parsing.TokenEventParser {
}
let mark = valueIndicator.endMark
events.append(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
}
@ -66,7 +66,7 @@ extension PureYAML.Parsing.TokenEventParser {
_ = try expect("indent") { $0.isIndent }
while isScalar(cursor.current) {
let token = cursor.advance() ?? first
let token = try cursor.advance() ?? first
let continuation = try plainContinuationText(from: token)
value += " " + continuation
endMark = token.endMark
@ -74,7 +74,7 @@ extension PureYAML.Parsing.TokenEventParser {
_ = try expect("dedent") { $0.isDedent }
let mark = properties.mark ?? first.mark
events.append(.scalar(
try emit(.scalar(
value: value,
anchor: properties.anchor,
tag: properties.tag,
@ -84,18 +84,18 @@ extension PureYAML.Parsing.TokenEventParser {
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: endMark)
}
mutating func consumeProperties() -> PureYAML.Parsing.NodeProperties {
mutating func consumeProperties() throws -> PureYAML.Parsing.NodeProperties {
var properties = PureYAML.Parsing.NodeProperties.none
while let token = cursor.current {
switch token.kind {
case let .anchor(anchor):
properties.anchor = anchor
properties.mark = properties.mark ?? token.mark
_ = cursor.advance()
_ = try cursor.advance()
case let .tag(tag):
properties.tag = tag
properties.mark = properties.mark ?? token.mark
_ = cursor.advance()
_ = try cursor.advance()
default:
return properties
}
@ -148,30 +148,30 @@ extension PureYAML.Parsing.TokenEventParser {
return current.mark.line == token.mark.line
}
func canParsePlainScalarWithContinuation(after token: PureYAML.Parsing.Token) -> Bool {
mutating func canParsePlainScalarWithContinuation(after token: PureYAML.Parsing.Token) throws -> Bool {
guard case .scalar(_, .plain) = cursor.current?.kind else {
return false
}
guard cursor.current?.mark.line == token.mark.line else {
return false
}
guard cursor.peek()?.kind.isIndent == true else {
guard try cursor.peek()?.kind.isIndent == true else {
return false
}
guard isScalar(cursor.peek(offset: 2)) else {
guard try isScalar(cursor.peek(offset: 2)) else {
return false
}
return true
}
func canParsePlainScalarContinuation() -> Bool {
mutating func canParsePlainScalarContinuation() throws -> Bool {
guard case .scalar(_, .plain) = cursor.current?.kind else {
return false
}
guard cursor.peek()?.kind.isIndent == true else {
guard try cursor.peek()?.kind.isIndent == true else {
return false
}
guard isScalar(cursor.peek(offset: 2)) else {
guard try isScalar(cursor.peek(offset: 2)) else {
return false
}
return true
@ -191,14 +191,14 @@ extension PureYAML.Parsing.TokenEventParser {
}
mutating func consumeMappingSiblingBoundaryDedents(minimumColumn: Int) throws {
while canConsumeMappingSiblingBoundaryDedent(minimumColumn: minimumColumn) {
while try canConsumeMappingSiblingBoundaryDedent(minimumColumn: minimumColumn) {
_ = try expect("dedent") { $0.isDedent }
}
}
func canConsumeMappingSiblingBoundaryDedent(minimumColumn: Int) -> Bool {
mutating func canConsumeMappingSiblingBoundaryDedent(minimumColumn: Int) throws -> Bool {
guard let width = dedentWidth(cursor.current),
let next = cursor.peek(),
let next = try cursor.peek(),
next.kind.isMappingKey,
next.mark.column >= minimumColumn
else {
@ -210,8 +210,8 @@ extension PureYAML.Parsing.TokenEventParser {
mutating func consumeMappingSiblingBoundaryReindent(minimumColumn: Int) throws {
guard let dedent = dedentWidth(cursor.current),
dedent < minimumColumn,
indentWidth(cursor.peek()) == minimumColumn,
let mappingKey = cursor.peek(offset: 2),
try indentWidth(cursor.peek()) == minimumColumn,
let mappingKey = try cursor.peek(offset: 2),
mappingKey.kind.isMappingKey,
mappingKey.mark.column == minimumColumn + 1
else {
@ -292,7 +292,7 @@ extension PureYAML.Parsing.TokenEventParser {
column: token.mark.column,
)
}
_ = cursor.advance()
_ = try cursor.advance()
return token
}

90
Sources/Parsing/TokenEventParser.swift
View File

@ -2,7 +2,8 @@ extension PureYAML.Parsing {
struct TokenEventParser {
var cursor: TokenCursor
let scalarParser: Parser
var events: [Event]
var events: [Event]?
var eventSink: ((Event) throws -> Void)?
init(
tokens: [Token],
@ -11,9 +12,41 @@ extension PureYAML.Parsing {
cursor = TokenCursor(tokens: tokens)
self.scalarParser = scalarParser
events = []
eventSink = nil
}
mutating func parse() throws -> [Event] {
try parseEvents()
return events ?? []
}
init(
tokens: [Token],
scalarParser: Parser,
eventSink: @escaping (Event) throws -> Void,
) {
cursor = TokenCursor(tokens: tokens)
self.scalarParser = scalarParser
events = nil
self.eventSink = eventSink
}
init(
yaml: String,
scalarParser: Parser,
eventSink: @escaping (Event) throws -> Void,
) {
cursor = TokenCursor(yaml: yaml)
self.scalarParser = scalarParser
events = nil
self.eventSink = eventSink
}
mutating func parseStreaming() throws {
try parseEvents()
}
mutating func parseEvents() throws {
let streamStart = try expect("stream start") { kind in
if case .streamStart = kind {
return true
@ -21,7 +54,7 @@ extension PureYAML.Parsing {
return false
}
events.append(.streamStart(mark: streamStart.mark))
try emit(.streamStart(mark: streamStart.mark))
var didParseDocument = false
while cursor.current?.kind.isStreamEnd != true {
try parseDocument()
@ -29,9 +62,9 @@ extension PureYAML.Parsing {
}
if !didParseDocument {
let mark = cursor.current?.mark ?? streamStart.endMark
events.append(.documentStart(mark: mark))
appendEmptyDocument(at: mark)
events.append(.documentEnd(mark: mark))
try emit(.documentStart(mark: mark))
try appendEmptyDocument(at: mark)
try emit(.documentEnd(mark: mark))
}
let streamEnd = try expect("stream end") { kind in
if case .streamEnd = kind {
@ -39,27 +72,34 @@ extension PureYAML.Parsing {
}
return false
}
events.append(.streamEnd(mark: streamEnd.mark))
return events
try emit(.streamEnd(mark: streamEnd.mark))
}
mutating func emit(_ event: Event) throws {
if let eventSink {
try eventSink(event)
} else {
events?.append(event)
}
}
}
}
extension PureYAML.Parsing.TokenEventParser {
mutating func parseDocument() throws {
let startMark = parseDocumentStartMark()
events.append(.documentStart(mark: startMark))
let startMark = try parseDocumentStartMark()
try emit(.documentStart(mark: startMark))
if cursor.current?.kind.isDocumentEnd == true {
let end = try expect("document end") { $0.isDocumentEnd }
appendEmptyDocument(at: end.mark)
events.append(.documentEnd(mark: end.endMark))
try appendEmptyDocument(at: end.mark)
try emit(.documentEnd(mark: end.endMark))
return
}
if cursor.current?.kind.isDocumentStart == true || cursor.current?.kind.isStreamEnd == true {
appendEmptyDocument(at: cursor.current?.mark ?? startMark)
events.append(.documentEnd(mark: cursor.current?.mark ?? startMark))
try appendEmptyDocument(at: cursor.current?.mark ?? startMark)
try emit(.documentEnd(mark: cursor.current?.mark ?? startMark))
return
}
@ -80,7 +120,7 @@ extension PureYAML.Parsing.TokenEventParser {
column: current.mark.column,
)
}
events.append(.documentEnd(mark: endMark))
try emit(.documentEnd(mark: endMark))
}
mutating func parseDocumentRootNode() throws -> PureYAML.Parsing.NodeResult {
@ -95,16 +135,16 @@ extension PureYAML.Parsing.TokenEventParser {
return result
}
mutating func parseDocumentStartMark() -> PureYAML.Parsing.Mark {
mutating func parseDocumentStartMark() throws -> PureYAML.Parsing.Mark {
guard cursor.current?.kind.isDocumentStart == true else {
return cursor.current?.mark ?? cursor.previous?.endMark ?? .start
}
let start = cursor.advance()
let start = try cursor.advance()
return start?.mark ?? .start
}
mutating func appendEmptyDocument(at mark: PureYAML.Parsing.Mark) {
events.append(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
mutating func appendEmptyDocument(at mark: PureYAML.Parsing.Mark) throws {
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark))
}
func canEndDocument(at token: PureYAML.Parsing.Token) -> Bool {
@ -112,7 +152,7 @@ extension PureYAML.Parsing.TokenEventParser {
}
mutating func parseNode() throws -> PureYAML.Parsing.NodeResult {
let properties = consumeProperties()
let properties = try consumeProperties()
return try parseNode(properties: properties)
}
@ -120,7 +160,7 @@ extension PureYAML.Parsing.TokenEventParser {
properties initialProperties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
var properties = initialProperties
properties.mergeUnset(from: consumeProperties())
try properties.mergeUnset(from: consumeProperties())
guard let token = cursor.current, !token.kind.isTerminator else {
let mark = properties.mark ?? cursor.current?.mark ?? cursor.previous?.endMark ?? .start
@ -129,9 +169,9 @@ extension PureYAML.Parsing.TokenEventParser {
switch token.kind {
case let .alias(anchor):
_ = cursor.advance()
_ = try cursor.advance()
let mark = properties.mark ?? token.mark
events.append(.alias(anchor: anchor, mark: mark))
try emit(.alias(anchor: anchor, mark: mark))
return PureYAML.Parsing.NodeResult(kind: .alias, endMark: token.endMark)
case .blockEntry:
return try parseBlockSequence(properties: properties)
@ -147,13 +187,13 @@ extension PureYAML.Parsing.TokenEventParser {
includeIndentedContinuation: false,
)
case let .scalar(value, style):
if canParsePlainScalarContinuation() {
if try canParsePlainScalarContinuation() {
return try parsePlainScalarWithContinuation(properties: properties)
}
_ = cursor.advance()
_ = try cursor.advance()
let decoded = try decodeScalar(value, style: style, line: token.mark.line)
let mark = properties.mark ?? token.mark
events.append(.scalar(
try emit(.scalar(
value: decoded,
anchor: properties.anchor,
tag: properties.tag,

65
Sources/Validation/DiagnosticValidation.swift
View File

@ -88,7 +88,7 @@ private extension PureYAML {
var diagnostics = preflightScanner.diagnostics(in: yaml, file: file)
do {
let documents = try parseStream(yaml)
let documents = try parseStreamForDiagnostics(yaml)
let result = validator.collect(documents)
diagnostics.append(contentsOf: result.issues.map { streamIssue in
Validation.Diagnostic(
@ -107,10 +107,11 @@ private extension PureYAML {
} catch let error as Parsing.ParseError {
diagnostics.append(Validation.Diagnostic(
kind: .parse,
code: error.diagnosticCode,
severity: .error,
file: file,
line: error.line,
column: error.column,
line: error.sourceLine,
column: error.sourceColumn,
reason: error.description,
))
return DiagnosticValidationResult(
@ -130,59 +131,11 @@ private extension PureYAML {
)
}
}
}
private extension PureYAML.Parsing.ParseError {
var line: Int? {
switch self {
case .emptyDocument:
nil
case let .tabIndentation(line),
let .unexpectedIndentation(line),
let .mixedCollectionStyles(line),
let .expectedMappingKey(line),
let .expectedAnchorName(line),
let .expectedAliasName(line),
let .incompatibleYAMLDirective(line),
let .unterminatedTag(line),
let .unterminatedQuotedString(line),
let .unsupportedDirective(_, line),
let .unsupportedMultiDocumentStream(line):
line
case let .expectedNode(line, _),
let .expectedScalarKey(line, _),
let .invalidTaggedScalar(_, _, line, _),
let .invalidMergeValue(line, _),
let .undefinedAlias(_, line, _),
let .unexpectedEvent(_, _, line, _),
let .unexpectedToken(_, _, line, _):
line
}
}
var column: Int? {
switch self {
case .emptyDocument,
.tabIndentation,
.unexpectedIndentation,
.mixedCollectionStyles,
.expectedMappingKey,
.expectedAnchorName,
.expectedAliasName,
.incompatibleYAMLDirective,
.unterminatedTag,
.unterminatedQuotedString,
.unsupportedDirective,
.unsupportedMultiDocumentStream:
nil
case let .expectedNode(_, column),
let .expectedScalarKey(_, column),
let .invalidTaggedScalar(_, _, _, column),
let .invalidMergeValue(_, column),
let .undefinedAlias(_, _, column),
let .unexpectedEvent(_, _, _, column),
let .unexpectedToken(_, _, _, column):
column
}
static func parseStreamForDiagnostics(_ yaml: String) throws -> [Stream.Document] {
let parser = Parsing.Parser()
let events = try parser.parseEvents(yaml)
var composer = Parsing.EventComposer(events: events, scalarParser: parser)
return try composer.composeStream()
}
}

7
Sources/Validation/PreflightScanner.swift
View File

@ -26,6 +26,7 @@ public extension PureYAML.Validation {
var diagnostics: [Diagnostic] = []
if let column = firstIndentationTabColumn(in: line) {
diagnostics.append(.preflight(
code: "tabIndentation",
file: file,
line: lineNumber,
column: column,
@ -34,6 +35,7 @@ public extension PureYAML.Validation {
}
if let column = missingMappingSpaceColumn(in: line) {
diagnostics.append(.preflight(
code: "missingMappingSpace",
file: file,
line: lineNumber,
column: column,
@ -42,6 +44,7 @@ public extension PureYAML.Validation {
}
if let column = missingSequenceSpaceColumn(in: line) {
diagnostics.append(.preflight(
code: "missingSequenceSpace",
file: file,
line: lineNumber,
column: column,
@ -50,6 +53,7 @@ public extension PureYAML.Validation {
}
if let column = invalidControlCharacterColumn(in: line) {
diagnostics.append(.preflight(
code: "invalidControlCharacter",
file: file,
line: lineNumber,
column: column,
@ -58,6 +62,7 @@ public extension PureYAML.Validation {
}
if let column = trailingWhitespaceColumn(in: line) {
diagnostics.append(.preflight(
code: "trailingWhitespace",
file: file,
line: lineNumber,
column: column,
@ -72,6 +77,7 @@ public extension PureYAML.Validation {
private extension PureYAML.Validation.Diagnostic {
static func preflight(
code: String,
file: String?,
line: Int,
column: Int,
@ -80,6 +86,7 @@ private extension PureYAML.Validation.Diagnostic {
) -> Self {
.init(
kind: .parse,
code: code,
severity: severity,
file: file,
line: line,

3
Sources/Validation/Validation.Diagnostic.swift
View File

@ -16,6 +16,7 @@ public extension PureYAML.Validation {
/// A parse or validation diagnostic suitable for batch validation reports.
struct Diagnostic: Equatable, Sendable, CustomStringConvertible {
public var kind: DiagnosticKind
public var code: String?
public var severity: Severity
public var file: String?
public var line: Int?
@ -26,6 +27,7 @@ public extension PureYAML.Validation {
public init(
kind: DiagnosticKind,
code: String? = nil,
severity: Severity,
file: String? = nil,
line: Int? = nil,
@ -35,6 +37,7 @@ public extension PureYAML.Validation {
reason: String,
) {
self.kind = kind
self.code = code
self.severity = severity
self.file = file
self.line = line

15
Sources/Validation/Validation.Report.swift
View File

@ -218,6 +218,9 @@ private extension PureYAML.Validation.Diagnostic {
.init(key: "severity", value: .string(severity.description)),
.init(key: "file", value: file.map(PureYAML.Model.Value.string) ?? .null),
]
if let code {
pairs.insert(.init(key: "code", value: .string(code)), at: 1)
}
if let line {
pairs.append(.init(key: "line", value: .int(line)))
}
@ -352,6 +355,18 @@ public extension PureYAML {
)
}
return Validation.Report(diagnostics)
} catch let error as PureYAML.Parsing.ParseError {
return Validation.Report([
Validation.Diagnostic(
kind: .parse,
code: error.diagnosticCode,
severity: .error,
file: file,
line: error.sourceLine,
column: error.sourceColumn,
reason: error.description,
),
])
} catch {
return Validation.Report([
Validation.Diagnostic(

145
Tests/ACMEZephyrFixtureTests.swift Normal file
View File

@ -0,0 +1,145 @@
import Foundation
@testable import PureYAML
import Testing
@Suite("ACME Fixtures")
struct ACMEZephyrFixtureTests {
@Test("Parses Stitcher-style ACME Zephyr source files")
func test_parsesStitcherStyleACMEZephyrSourceFiles() throws {
let spec = try loadFixture("spec.yml", subdirectory: "Fixtures/acme/zephyr-data")
let activeFeature = try loadFixture(
"activeZorplexFeature.yml",
subdirectory: "Fixtures/acme/zephyr-data/schemas",
)
let featureEnum = try loadFixture("zorplexFeature.yml", subdirectory: "Fixtures/acme/zephyr-data/schemas")
let specRoot = try requireMapping(PureYAML.parse(spec))
#expect(specRoot["openapi"] == .string("3.0.3"))
let schemas = expectMapping(expectMapping(specRoot["components"])?["schemas"])
#expect(expectMapping(schemas?["ActiveZorplexFeature"])?["$ref"] == .string("./schemas/activeZorplexFeature.yml"))
#expect(expectMapping(schemas?["ZorplexFeature"])?["$ref"] == .string("./schemas/zorplexFeature.yml"))
let activeRoot = try requireMapping(PureYAML.parse(activeFeature))
#expect(activeRoot["type"] == .string("object"))
#expect(requireSequence(activeRoot["required"])?.contains(.string("createdBy")) == true)
let properties = expectMapping(activeRoot["properties"])
#expect(expectMapping(properties?["feature"])?["$ref"] == .string("./zorplexFeature.yml"))
#expect(expectMapping(properties?["deletedAt"])?["nullable"] == .bool(true))
let featureRoot = try requireMapping(PureYAML.parse(featureEnum))
#expect(featureRoot["type"] == .string("string"))
#expect(requireSequence(featureRoot["enum"])?.contains(.string("QUASAR_MATRIX_V2")) == true)
}
@Test("Parses bundled ACME Zephyr output")
func test_parsesBundledACMEZephyrOutput() throws {
let bundled = try loadFixture("bundled.yml", subdirectory: "Fixtures/acme/zephyr-data")
let root = try requireMapping(PureYAML.parse(bundled))
#expect(root["openapi"] == .string("3.0.3"))
let paths = expectMapping(root["paths"])
let operation = expectMapping(expectMapping(paths?["/activeZorplexFeatures"])?["get"])
#expect(operation?["operationId"] == .string("getActiveZorplexFeatures"))
let components = expectMapping(root["components"])
let schemas = expectMapping(components?["schemas"])
let active = expectMapping(schemas?["ActiveZorplexFeature"])
let properties = expectMapping(active?["properties"])
let feature = expectMapping(properties?["feature"])
#expect(requireSequence(feature?["enum"])?.contains(.string("BLARF_OPTIMIZATION")) == true)
#expect(expectMapping(properties?["deletedAt"])?["nullable"] == .bool(true))
}
@Test("Parses renamed Birch-style ACME catalog representative files")
func test_parsesRenamedBirchStyleACMECatalogRepresentativeFiles() throws {
let eventSpec = try loadFixture("spec.yml", subdirectory: "Fixtures/acme/birch-catalog/ui-event")
let originatorSpec = try loadFixture("spec.yml", subdirectory: "Fixtures/acme/birch-catalog/originator")
let originatorOffer = try loadFixture("offer.yml", subdirectory: "Fixtures/acme/birch-catalog/originator/schemas")
let apiError = try loadFixture("apiError.yml", subdirectory: "Fixtures/acme/birch-catalog/core/schemas")
let eventRoot = try requireMapping(PureYAML.parse(eventSpec))
#expect(eventRoot["openapi"] == .string("3.0.3"))
let eventSecurity = expectMapping(expectMapping(eventRoot["components"])?["securitySchemes"])
let oauth = expectMapping(eventSecurity?["oauth"])
let scopes = expectMapping(expectMapping(expectMapping(oauth?["flows"])?["clientCredentials"])?["scopes"])
#expect(scopes?["ui-event:write"] == .string("Allows recording UI events"))
let eventSchemas = expectMapping(expectMapping(eventRoot["components"])?["schemas"])
#expect(expectMapping(eventSchemas?["ApiError"])?["$ref"] == .string("../core/schemas/apiError.yml"))
#expect(expectMapping(eventSchemas?["SessionInit"])?["$ref"] == .string("./schemas/sessionInit.yml"))
let offerDisplays = expectMapping(expectMapping(eventRoot["paths"])?["/offerDisplays"])
let createOperation = expectMapping(offerDisplays?["post"])
#expect(expectMapping(createOperation?["requestBody"])?["$ref"] == .string("./request-bodies/offerDisplay.yml"))
let originatorRoot = try requireMapping(PureYAML.parse(originatorSpec))
let originatorSchemas = expectMapping(expectMapping(originatorRoot["components"])?["schemas"])
#expect(expectMapping(originatorSchemas?["ApiError"])?["$ref"] == .string("../core/schemas/apiError.yml"))
#expect(expectMapping(originatorSchemas?["CreditCardOfferRecord"])?["$ref"] == .string("./schemas/creditCardOfferRecord.yml"))
let originatorSecurity = expectMapping(expectMapping(originatorRoot["components"])?["securitySchemes"])
let originatorOAuth = expectMapping(originatorSecurity?["oauth"])
let originatorClientCredentials = expectMapping(expectMapping(originatorOAuth?["flows"])?["clientCredentials"])
let originatorScopes = expectMapping(originatorClientCredentials?["scopes"])
#expect(originatorScopes?["originator:admin:write"] == .string("Allows write access to all admin endpoints in originator"))
let offerRoot = try requireMapping(PureYAML.parse(originatorOffer))
let offerProperties = expectMapping(offerRoot["properties"])
#expect(expectMapping(offerProperties?["partner"])?["$ref"] == .string("./demandPartner.yml"))
#expect(expectMapping(offerProperties?["productType"])?["$ref"] == .string("../../core/schemas/productType.yml"))
#expect(expectMapping(offerProperties?["productSubType"])?["$ref"] == .string("../../lead/schemas/productSubType.yml"))
let errorRoot = try requireMapping(PureYAML.parse(apiError))
let errorProperties = expectMapping(errorRoot["properties"])
#expect(expectMapping(errorProperties?["details"])?["$ref"] == .string("./apiErrorDetails.yml"))
#expect(expectMapping(errorProperties?["message"])?["readOnly"] == .bool(true))
}
@Test("Parses all renamed Birch-style ACME catalog YAML files")
func test_parsesAllRenamedBirchStyleACMECatalogYAMLFiles() throws {
let rootURL = try #require(Bundle.module.url(
forResource: "birch-catalog",
withExtension: nil,
subdirectory: "Fixtures/acme",
))
let files = try FileManager.default
.subpathsOfDirectory(atPath: rootURL.path)
.filter { $0.hasSuffix(".yml") || $0.hasSuffix(".yaml") }
.sorted()
#expect(files.count == 594)
for file in files {
let url = rootURL.appendingPathComponent(file)
let source = try String(contentsOf: url, encoding: .utf8)
_ = try PureYAML.parse(source)
}
}
private func loadFixture(_ name: String, subdirectory: String) throws -> String {
let url = try #require(Bundle.module.url(forResource: name, withExtension: nil, subdirectory: subdirectory))
return try String(contentsOf: url, encoding: .utf8)
}
private func requireMapping(_ value: PureYAML.Model.Value) throws -> PureYAML.Model.Mapping {
guard case let .mapping(mapping) = value else {
recordIssue("expected mapping")
return .init()
}
return mapping
}
private func expectMapping(_ value: PureYAML.Model.Value?) -> PureYAML.Model.Mapping? {
guard case let .mapping(mapping)? = value else {
recordIssue("expected mapping")
return nil
}
return mapping
}
private func requireSequence(_ value: PureYAML.Model.Value?) -> [PureYAML.Model.Value]? {
guard case let .sequence(sequence)? = value else {
recordIssue("expected sequence")
return nil
}
return sequence
}
}

61
Tests/DiagnosticValidationReportTests.swift
View File

@ -37,6 +37,7 @@ struct DiagnosticValidationReportTests {
#expect(report.diagnostics == [
.init(
kind: .parse,
code: "trailingWhitespace",
severity: .warning,
file: "duplicates.yaml",
line: 3,
@ -71,6 +72,7 @@ struct DiagnosticValidationReportTests {
#expect(report.diagnostics == [
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "broken.yaml",
line: 2,
@ -79,6 +81,7 @@ struct DiagnosticValidationReportTests {
),
.init(
kind: .parse,
code: "tabIndentation",
severity: .error,
file: "broken.yaml",
line: 3,
@ -87,6 +90,7 @@ struct DiagnosticValidationReportTests {
),
.init(
kind: .parse,
code: "unterminatedQuotedString",
severity: .error,
file: "broken.yaml",
line: 1,
@ -118,6 +122,7 @@ struct DiagnosticValidationReportTests {
#expect(error.report.diagnostics.count == 3)
#expect(json.contains(#""title": "Production YAML Validation""#))
#expect(json.contains(#""code": "missingMappingSpace""#))
#expect(json.contains(#""line": 2"#))
#expect(json.contains(#""column": 9"#))
#expect(json.contains(#""reason": "missing space after ':' in mapping entry""#))
@ -131,6 +136,7 @@ struct DiagnosticValidationReportTests {
#expect(yamlRoot?["summary"]?.mapping?["diagnostics"] == PureYAML.Model.Value.int(3))
#expect(yamlRoot?["summary"]?.mapping?["errors"] == PureYAML.Model.Value.int(3))
#expect(yamlRoot?["summary"]?.mapping?["warnings"] == PureYAML.Model.Value.int(0))
#expect(firstDiagnostic?["code"] == PureYAML.Model.Value.string("missingMappingSpace"))
#expect(firstDiagnostic?["line"] == PureYAML.Model.Value.int(2))
#expect(firstDiagnostic?["column"] == PureYAML.Model.Value.int(9))
#expect(firstDiagnostic?["reason"] == PureYAML.Model.Value.string("missing space after ':' in mapping entry"))
@ -157,6 +163,7 @@ struct DiagnosticValidationReportTests {
#expect(report.diagnostics == [
.init(
kind: .parse,
code: "trailingWhitespace",
severity: .warning,
file: "warning.yaml",
line: 1,
@ -165,4 +172,58 @@ struct DiagnosticValidationReportTests {
),
])
}
@Test("Malformed production-shaped YAML reports ten or more structured diagnostics")
func test_malformedProductionShapedYAMLReportsTenOrMoreStructuredDiagnostics() {
let report = PureYAML.diagnosticValidationReport(
[
"apiVersion:v1",
"\tmetadata:",
"-bad",
"name: demo ",
"image:nginx",
"\t- item",
"-item",
"version:1",
"ports: [80]\t",
"owner:team",
"title: \"open",
].joined(separator: "\n"),
file: "production-broken.yaml",
)
#expect(report.diagnostics.count == 11)
#expect(report.errors.count == 9)
#expect(report.warnings.count == 2)
#expect(report.diagnostics.map(\.code) == [
"missingMappingSpace",
"tabIndentation",
"missingSequenceSpace",
"trailingWhitespace",
"missingMappingSpace",
"tabIndentation",
"missingSequenceSpace",
"missingMappingSpace",
"trailingWhitespace",
"missingMappingSpace",
"unterminatedQuotedString",
])
#expect(report.diagnostics.first == .init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production-broken.yaml",
line: 1,
column: 12,
reason: "missing space after ':' in mapping entry",
))
#expect(report.diagnostics.last == .init(
kind: .parse,
code: "unterminatedQuotedString",
severity: .error,
file: "production-broken.yaml",
line: 11,
reason: "unterminated quoted string at line 11",
))
}
}

4
Tests/DumperTests.swift
View File

@ -28,11 +28,13 @@ struct DumperTests {
let value = PureYAML.Model.Value.mapping(.init([
.init(key: "a:b", value: .string("quote \" slash \\ tab\t")),
.init(key: "hash#key", value: .string("line\nnext")),
.init(key: "description ", value: .string("keeps trailing key space")),
]))
#expect(PureYAML.dump(value) == """
"a:b": "quote \\" slash \\\\ tab\\t"
a:b: "quote \\" slash \\\\ tab\\t"
"hash#key": "line\\nnext"
"description ": "keeps trailing key space"
""")
#expect(!PureYAML.dump(value).contains("quote \" slash \\ tab\t"))

167
Tests/EventStreamingParserTests.swift Normal file
View File

@ -0,0 +1,167 @@
@testable import PureYAML
import Testing
@Suite("Event Streaming Parser")
struct EventStreamingParserTests {
@Test(
"Streaming composer matches full event composer for single documents",
arguments: EventStreamingFixtures.singleDocuments,
)
func test_streamingComposerMatchesFullEventComposer(testCase: EventStreamingFixtures.DocumentCase) throws {
let parser = PureYAML.Parsing.Parser()
let streaming = try parser.parse(testCase.yaml)
let fullEvents = try composeSingleDocumentWithFullEvents(testCase.yaml)
#expect(streaming == fullEvents)
}
@Test(
"Streaming composer matches full event composer for streams",
arguments: EventStreamingFixtures.streams,
)
func test_streamingComposerMatchesFullEventComposerForStreams(testCase: EventStreamingFixtures.DocumentCase) throws {
let parser = PureYAML.Parsing.Parser()
let streaming = try parser.parseStream(testCase.yaml)
let fullEvents = try composeStreamWithFullEvents(testCase.yaml)
#expect(streaming == fullEvents)
}
@Test("Streaming parser preserves duplicate keys for validation")
func test_streamingParserPreservesDuplicateKeysForValidation() throws {
let value = try PureYAML.parse("""
root:
title: First
title: Second
flow: {id: one, id: two}
""")
let issues = PureYAML.Validation.Validator().collect(value).issues
#expect(issues.map(\.description) == [
"error: Duplicate mapping key 'title' at $.root.title",
"error: Duplicate mapping key 'id' at $.flow.id",
])
}
@Test(
"Lazy token cursor matches full scanner tokens",
arguments: EventStreamingFixtures.singleDocuments + EventStreamingFixtures.streams,
)
func test_lazyTokenCursorMatchesFullScannerTokens(testCase: EventStreamingFixtures.DocumentCase) throws {
let full = try PureYAML.Parsing.Scanner().scan(testCase.yaml)
.filter { !$0.kind.isComment }
let lazy = try collectLazyTokens(testCase.yaml)
#expect(lazy == full)
}
}
enum EventStreamingFixtures {
struct DocumentCase: CustomTestStringConvertible {
var name: String
var yaml: String
var testDescription: String {
name
}
}
static let singleDocuments = [
DocumentCase(
name: "block mappings and sequences",
yaml: """
servers:
- url: /
description: Default
""",
),
DocumentCase(
name: "anchors aliases and merge keys",
yaml: """
defaults: &defaults
enabled: true
retries: 3
service:
<<: *defaults
name: API
""",
),
DocumentCase(
name: "flow collections and tags",
yaml: """
flow: [!Thing &item "value", *item, {tagged: !<tag:example.com,2026:thing> 'it''s'}]
""",
),
DocumentCase(
name: "block scalars",
yaml: """
text: |
one
two
folded: >
three
four
""",
),
DocumentCase(
name: "JSON style input",
yaml: #"{"openapi":"3.0.0","info":{"title":"JSON API"},"items":["a:b","c"],"enabled":true}"#,
),
DocumentCase(
name: "complex mapping key",
yaml: """
? {name: service, version: 1}
: active
""",
),
]
static let streams = [
DocumentCase(
name: "multi document sequence stream",
yaml: """
---
- one
- two
---
name: three
...
---
null
""",
),
DocumentCase(
name: "empty explicit documents",
yaml: """
---
...
---
""",
),
]
}
private func composeSingleDocumentWithFullEvents(_ yaml: String) throws -> PureYAML.Model.Value {
let parser = PureYAML.Parsing.Parser()
let events = try parser.parseEvents(yaml)
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: parser)
return try composer.compose()
}
private func composeStreamWithFullEvents(_ yaml: String) throws -> [PureYAML.Stream.Document] {
let parser = PureYAML.Parsing.Parser()
let events = try parser.parseEvents(yaml)
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: parser)
return try composer.composeStream()
}
private func collectLazyTokens(_ yaml: String) throws -> [PureYAML.Parsing.Token] {
var cursor = PureYAML.Parsing.TokenCursor(yaml: yaml)
var tokens: [PureYAML.Parsing.Token] = []
while let token = cursor.current {
tokens.append(token)
_ = try cursor.advance()
}
return tokens
}

171
Tests/JSONCompatibilityTests.swift Normal file
View File

@ -0,0 +1,171 @@
import Foundation
@testable import PureYAML
import Testing
@Suite("JSON Compatibility")
struct JSONCompatibilityTests {
@Test(
"Parses valid JSON as YAML 1.2",
arguments: JSONCompatibilityFixtures.validDocuments,
)
func test_parsesValidJSONAsYAML(testCase: JSONCompatibilityFixtures.ValidDocument) throws {
let json = try JSONOracle.decode(testCase.json)
let yaml = try JSONOracle(pureYAML: PureYAML.parse(testCase.json))
#expect(yaml == json)
}
@Test(
"JSON oracle rejects invalid JSON fixtures",
arguments: JSONCompatibilityFixtures.invalidDocuments,
)
func test_jsonOracleRejectsInvalidJSON(testCase: JSONCompatibilityFixtures.InvalidDocument) {
do {
_ = try JSONOracle.decode(testCase.json)
recordIssue("expected JSON oracle to reject \(testCase.name)")
} catch {}
}
}
enum JSONCompatibilityFixtures {
struct ValidDocument: CustomTestStringConvertible {
var name: String
var json: String
var testDescription: String {
name
}
}
struct InvalidDocument: CustomTestStringConvertible {
var name: String
var json: String
var testDescription: String {
name
}
}
static let validDocuments = [
ValidDocument(
name: "compact object",
json: #"{"openapi":"3.0.0","info":{"title":"JSON API","version":"1"},"paths":{},"enabled":true,"count":3}"#,
),
ValidDocument(
name: "array root",
json: #"[{"a":1},{"b":null},{"items":[true,false]}]"#,
),
ValidDocument(
name: "scalar roots",
json: #""hello""#,
),
ValidDocument(
name: "integer root",
json: #"123"#,
),
ValidDocument(
name: "boolean root",
json: #"true"#,
),
ValidDocument(
name: "null root",
json: #"null"#,
),
ValidDocument(
name: "string escapes",
json: #"{"quote":"\"","slash":"\\","solidus":"\/","backspace":"\b","formfeed":"\f","newline":"line\nnext","return":"a\rb","tab":"a\tb"}"#,
),
ValidDocument(
name: "unicode escapes",
json: #"{"bmp":"\u263A","surrogate":"\uD834\uDD1E"}"#,
),
ValidDocument(
name: "number forms",
json: #"{"zero":0,"negative":-3,"fraction":1.25,"small":1e-6,"big":6.02E23}"#,
),
ValidDocument(
name: "insignificant whitespace",
json: """
{
"items" : [
{ "url" : "https://api.example.com/a:b" },
{ "window" : "10:30" }
]
}
""",
),
ValidDocument(
name: "empty containers",
json: #"{"object":{},"array":[]}"#,
),
]
static let invalidDocuments = [
InvalidDocument(name: "missing value", json: #"{"a":}"#),
InvalidDocument(name: "double comma", json: #"[1,,2]"#),
InvalidDocument(name: "unquoted key", json: #"{a:1}"#),
InvalidDocument(name: "comment", json: #"{"a":1 // no comments in JSON"#),
InvalidDocument(name: "single quoted string", json: #"{'a':1}"#),
]
}
enum JSONOracle: Equatable, Decodable {
case null
case bool(Bool)
case int(Int)
case double(Double)
case string(String)
case array([JSONOracle])
case object([String: JSONOracle])
init(from decoder: Decoder) throws {
let container = try decoder.singleValueContainer()
if container.decodeNil() {
self = .null
} else if let value = try? container.decode(Bool.self) {
self = .bool(value)
} else if let value = try? container.decode(Int.self) {
self = .int(value)
} else if let value = try? container.decode(Double.self) {
self = .double(value)
} else if let value = try? container.decode(String.self) {
self = .string(value)
} else if let value = try? container.decode([JSONOracle].self) {
self = .array(value)
} else {
self = try .object(container.decode([String: JSONOracle].self))
}
}
init(pureYAML value: PureYAML.Model.Value) throws {
switch value {
case .null:
self = .null
case let .bool(value):
self = .bool(value)
case let .int(value):
self = .int(value)
case let .double(value):
self = .double(value)
case let .string(value):
self = .string(value)
case let .sequence(values):
self = try .array(values.map(JSONOracle.init(pureYAML:)))
case let .mapping(mapping):
self = try .object(mapping.pairs.reduce(into: [:]) { result, pair in
guard case let .string(key) = pair.keyNode.value else {
throw JSONOracleFailure.nonStringKey
}
result[key] = try JSONOracle(pureYAML: pair.value)
})
}
}
static func decode(_ json: String) throws -> JSONOracle {
try JSONDecoder().decode(JSONOracle.self, from: Data(json.utf8))
}
}
enum JSONOracleFailure: Error {
case nonStringKey
}

25
Tests/ParsingCompatibilityTests.swift
View File

@ -112,6 +112,31 @@ struct ParsingCompatibilityTests {
#expect(root?["missing"] == nil)
}
@Test("Parses JSON-style compact flow mappings")
func test_jsonStyleCompactFlowMappings() throws {
let root = try requireMapping(PureYAML.parse(
#"{"openapi":"3.0.0","info":{"title":"JSON API","version":"1"},"paths":{},"enabled":true,"count":3}"#,
))
#expect(root?["openapi"] == .string("3.0.0"))
#expect(root?.mapping("info")?["title"] == .string("JSON API"))
#expect(root?.mapping("info")?["version"] == .string("1"))
#expect(root?["paths"] == .mapping(.init()))
#expect(root?["enabled"] == .bool(true))
#expect(root?["count"] == .int(3))
}
@Test("Keeps colons inside JSON-style flow mapping values")
func test_jsonStyleFlowMappingValuesCanContainColons() throws {
let root = try requireMapping(PureYAML.parse(
#"{"url":"https://api.example.com/v1","window":"10:30","items":["a:b","c"]}"#,
))
#expect(root?["url"] == .string("https://api.example.com/v1"))
#expect(root?["window"] == .string("10:30"))
#expect(root?["items"] == .sequence([.string("a:b"), .string("c")]))
}
@Test(
"Resolves selected Yams-compatible scalar spellings",
arguments: ScalarCompatibilityFixtures.resolvedScalars,

298
Tests/PerformancePhaseTests.swift Normal file
View File

@ -0,0 +1,298 @@
import Foundation
@testable import PureYAML
import Testing
@Suite("Performance Phases")
struct PerformancePhaseTests {
@Test("Profiles parser phases for representative slow corpus seeds")
func test_profilesParserPhasesForRepresentativeSeeds() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_RUN_PHASE_PROFILER"] == "1" else {
return
}
let measurements = try PerformancePhaseSeed.all.map(profile)
let slowestLazyParse = measurements.map(\.lazyParse.seconds).max() ?? 0
let slowestMaterializedPipeline = measurements.map(\.materializedPipelineSeconds).max() ?? 0
let report: [String: Any] = [
"summary": [
"seeds": measurements.count,
"slowestLazyParseSeconds": slowestLazyParse,
"slowestMaterializedPipelineSeconds": slowestMaterializedPipeline,
],
"measurements": measurements.map(\.object),
]
let artifactURL = try performancePhaseArtifactURL()
let data = try JSONSerialization.data(withJSONObject: report, options: [.prettyPrinted, .sortedKeys])
try data.write(to: artifactURL)
print("PureYAML phase profiler seeds: \(measurements.count)")
for measurement in measurements {
print(measurement.summaryLine)
}
print("Report: \(artifactURL.path)")
}
}
private struct PerformancePhaseSeed {
var id: String
var localPath: String
var iterations: Int
static let all = [
PerformancePhaseSeed(
id: "cert-manager-crd",
localPath: "cert-manager-certificate-crd.yaml",
iterations: 20,
),
PerformancePhaseSeed(
id: "stripe-yaml",
localPath: "api-guru-stripe-openapi.yaml",
iterations: 2,
),
PerformancePhaseSeed(
id: "zoom-yaml",
localPath: "api-guru-zoom-openapi.yaml",
iterations: 2,
),
]
}
private struct PhaseTiming {
var seconds: TimeInterval
var units: Int
var object: [String: Any] {
[
"seconds": seconds,
"units": units,
]
}
}
private struct PhaseMeasurement {
var seed: PerformancePhaseSeed
var bytes: Int
var scanner: PhaseTiming
var eventParser: PhaseTiming
var composer: PhaseTiming
var lazyParse: PhaseTiming
var validation: PhaseTiming
var dumping: PhaseTiming
var materializedPipelineSeconds: TimeInterval {
scanner.seconds + eventParser.seconds + composer.seconds
}
var object: [String: Any] {
[
"id": seed.id,
"localPath": seed.localPath,
"bytes": bytes,
"iterations": seed.iterations,
"scanner": scanner.object,
"eventParser": eventParser.object,
"composer": composer.object,
"lazyParse": lazyParse.object,
"validation": validation.object,
"dumping": dumping.object,
"materializedPipelineSeconds": materializedPipelineSeconds,
]
}
var summaryLine: String {
[
seed.id,
"bytes=\(bytes)",
"iter=\(seed.iterations)",
"scan=\(format(scanner.seconds))s",
"events=\(format(eventParser.seconds))s",
"compose=\(format(composer.seconds))s",
"lazy=\(format(lazyParse.seconds))s",
"validate=\(format(validation.seconds))s",
"dump=\(format(dumping.seconds))s",
].joined(separator: " ")
}
}
private struct WarmedPerformanceInputs {
var tokens: [PureYAML.Parsing.Token]
var events: [PureYAML.Parsing.Event]
var documents: [PureYAML.Stream.Document]
}
private func profile(_ seed: PerformancePhaseSeed) throws -> PhaseMeasurement {
let yaml = try String(contentsOf: fixtureURL(for: seed), encoding: .utf8)
let parser = PureYAML.Parsing.Parser()
let warmed = try warmup(yaml: yaml, parser: parser)
let scanner = try timeScanner(yaml: yaml, iterations: seed.iterations)
let eventParser = try timeEventParser(tokens: warmed.tokens, parser: parser, iterations: seed.iterations)
let composer = try timeComposer(events: warmed.events, parser: parser, iterations: seed.iterations)
let lazyParse = try timeLazyParse(yaml: yaml, iterations: seed.iterations)
let validation = timeValidation(documents: warmed.documents, iterations: seed.iterations)
let dumping = timeDumping(documents: warmed.documents, iterations: seed.iterations)
return PhaseMeasurement(
seed: seed,
bytes: yaml.utf8.count,
scanner: .init(seconds: scanner.seconds, units: scanner.value),
eventParser: .init(seconds: eventParser.seconds, units: eventParser.value),
composer: .init(seconds: composer.seconds, units: composer.value),
lazyParse: .init(seconds: lazyParse.seconds, units: lazyParse.value),
validation: .init(seconds: validation.seconds, units: validation.value),
dumping: .init(seconds: dumping.seconds, units: dumping.value),
)
}
private func warmup(
yaml: String,
parser: PureYAML.Parsing.Parser,
) throws -> WarmedPerformanceInputs {
let tokens = try PureYAML.Parsing.Scanner().scan(yaml)
var eventParser = PureYAML.Parsing.TokenEventParser(tokens: tokens, scalarParser: parser)
let events = try eventParser.parse()
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: parser)
let documents = try composer.composeStream()
_ = PureYAML.Validation.Validator().collect(documents)
_ = PureYAML.dump(documents)
_ = try PureYAML.parseStream(yaml)
return WarmedPerformanceInputs(tokens: tokens, events: events, documents: documents)
}
private func timeScanner(
yaml: String,
iterations: Int,
) throws -> (seconds: TimeInterval, value: Int) {
try timed {
var units = 0
for _ in 0 ..< iterations {
units += try PureYAML.Parsing.Scanner().scan(yaml).count
}
return units
}
}
private func timeEventParser(
tokens: [PureYAML.Parsing.Token],
parser: PureYAML.Parsing.Parser,
iterations: Int,
) throws -> (seconds: TimeInterval, value: Int) {
try timed {
var units = 0
for _ in 0 ..< iterations {
var eventParser = PureYAML.Parsing.TokenEventParser(tokens: tokens, scalarParser: parser)
units += try eventParser.parse().count
}
return units
}
}
private func timeComposer(
events: [PureYAML.Parsing.Event],
parser: PureYAML.Parsing.Parser,
iterations: Int,
) throws -> (seconds: TimeInterval, value: Int) {
try timed {
var units = 0
for _ in 0 ..< iterations {
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: parser)
units += try composer.composeStream().count
}
return units
}
}
private func timeLazyParse(
yaml: String,
iterations: Int,
) throws -> (seconds: TimeInterval, value: Int) {
try timed {
var units = 0
for _ in 0 ..< iterations {
units += try PureYAML.parseStream(yaml).count
}
return units
}
}
private func timeValidation(
documents: [PureYAML.Stream.Document],
iterations: Int,
) -> (seconds: TimeInterval, value: Int) {
timed {
var units = 0
for _ in 0 ..< iterations {
units += PureYAML.Validation.Validator().collect(documents).issues.count
}
return units
}
}
private func timeDumping(
documents: [PureYAML.Stream.Document],
iterations: Int,
) -> (seconds: TimeInterval, value: Int) {
timed {
var units = 0
for _ in 0 ..< iterations {
units += PureYAML.dump(documents).utf8.count
}
return units
}
}
private func fixtureURL(for seed: PerformancePhaseSeed) throws -> URL {
try #require(Bundle.module.url(
forResource: seed.resourceName,
withExtension: seed.resourceExtension,
subdirectory: "Fixtures/real-yaml",
))
}
private func performancePhaseArtifactURL() throws -> URL {
if let path = ProcessInfo.processInfo.environment["PUREYAML_PHASE_PROFILER_ARTIFACT"] {
let url = URL(fileURLWithPath: path)
try FileManager.default.createDirectory(
at: url.deletingLastPathComponent(),
withIntermediateDirectories: true,
)
return url
}
let url = URL(fileURLWithPath: FileManager.default.currentDirectoryPath)
.appendingPathComponent(".build")
.appendingPathComponent("pureyaml-artifacts")
.appendingPathComponent("performance")
.appendingPathComponent("phase-profile.json")
try FileManager.default.createDirectory(
at: url.deletingLastPathComponent(),
withIntermediateDirectories: true,
)
return url
}
private func timed(_ body: () throws -> Int) rethrows -> (seconds: TimeInterval, value: Int) {
let start = DispatchTime.now().uptimeNanoseconds
let value = try body()
let end = DispatchTime.now().uptimeNanoseconds
return (Double(end - start) / 1_000_000_000, value)
}
private func format(_ value: Double) -> String {
String(format: "%.3f", value)
}
private extension PerformancePhaseSeed {
var resourceName: String {
guard let dotIndex = localPath.lastIndex(of: ".") else {
return localPath
}
return String(localPath[..<dotIndex])
}
var resourceExtension: String {
guard let dotIndex = localPath.lastIndex(of: ".") else {
return ""
}
return String(localPath[localPath.index(after: dotIndex)...])
}
}

26
Tests/PreflightScannerTests.swift
View File

@ -56,6 +56,7 @@ let preflightLineCases: [PreflightLineCase] = [
lineNumber: 7,
expected: .init(
kind: .parse,
code: "tabIndentation",
severity: .error,
file: "input.yaml",
line: 7,
@ -69,6 +70,7 @@ let preflightLineCases: [PreflightLineCase] = [
lineNumber: 8,
expected: .init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "input.yaml",
line: 8,
@ -82,6 +84,7 @@ let preflightLineCases: [PreflightLineCase] = [
lineNumber: 9,
expected: .init(
kind: .parse,
code: "missingSequenceSpace",
severity: .error,
file: "input.yaml",
line: 9,
@ -95,6 +98,7 @@ let preflightLineCases: [PreflightLineCase] = [
lineNumber: 10,
expected: .init(
kind: .parse,
code: "trailingWhitespace",
severity: .warning,
file: "input.yaml",
line: 10,
@ -108,6 +112,7 @@ let preflightLineCases: [PreflightLineCase] = [
lineNumber: 11,
expected: .init(
kind: .parse,
code: "invalidControlCharacter",
severity: .error,
file: "input.yaml",
line: 11,
@ -154,6 +159,7 @@ let productionLikeFaultyYAML = [
let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
.init(
kind: .parse,
code: "tabIndentation",
severity: .error,
file: "production.yaml",
line: 1,
@ -162,6 +168,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 2,
@ -170,6 +177,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 6,
@ -178,6 +186,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingSequenceSpace",
severity: .error,
file: "production.yaml",
line: 9,
@ -186,6 +195,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 10,
@ -194,6 +204,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingSequenceSpace",
severity: .error,
file: "production.yaml",
line: 12,
@ -202,6 +213,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 14,
@ -210,6 +222,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 15,
@ -218,6 +231,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "trailingWhitespace",
severity: .warning,
file: "production.yaml",
line: 16,
@ -226,6 +240,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "invalidControlCharacter",
severity: .error,
file: "production.yaml",
line: 17,
@ -234,6 +249,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 21,
@ -242,6 +258,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingSequenceSpace",
severity: .error,
file: "production.yaml",
line: 22,
@ -250,6 +267,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 23,
@ -258,6 +276,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 24,
@ -266,6 +285,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 25,
@ -274,6 +294,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 26,
@ -282,6 +303,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 27,
@ -290,6 +312,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 28,
@ -298,6 +321,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 29,
@ -306,6 +330,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 30,
@ -314,6 +339,7 @@ let productionLikeFaultyDiagnostics: [PureYAML.Validation.Diagnostic] = [
),
.init(
kind: .parse,
code: "missingMappingSpace",
severity: .error,
file: "production.yaml",
line: 31,

217
Tests/RealYAMLCorpusPropertyTests.swift Normal file
View File

@ -0,0 +1,217 @@
import Foundation
@testable import PureYAML
import Testing
@Suite("Real YAML Corpus Properties")
struct RealYAMLCorpusPropertyTests {
@Test("Generated valid YAML documents parse dump and parse stably")
func test_generatedValidYAMLDocumentsParseDumpAndParseStably() throws {
var generator = DeterministicGenerator(seed: 0x5055_5941_4D4C)
for index in 0 ..< 100 {
let yaml = generator.validYAMLDocument(index: index)
let documents = try PureYAML.parseStream(yaml)
try PureYAML.validate(documents)
let dumped = PureYAML.dump(documents)
let reparsed = try PureYAML.parseStream(dumped)
#expect(reparsed == documents, "generated case \(index) changed after parse/dump/parse")
}
}
@Test("Real seed mutations return structured diagnostics without crashing")
func test_realSeedMutationsReturnStructuredDiagnosticsWithoutCrashing() throws {
let seeds = try PropertyCorpusSeed.loadAll()
.filter { $0.tier == "default" }
.prefix(10)
for seed in seeds {
let source = try seed.load()
for mutation in PropertyMutation.allCases {
let mutated = mutation.apply(to: source, seedID: seed.id)
let report = PureYAML.diagnosticValidationReport(mutated, file: "\(seed.id)-\(mutation.rawValue).yaml")
#expect(!report.diagnostics.isEmpty, "\(seed.id) \(mutation.rawValue) produced no diagnostics")
#expect(report.diagnostics.allSatisfy { $0.kind == .parse })
let codes = report.diagnostics.map { $0.code ?? "nil" }.joined(separator: ",")
#expect(
report.diagnostics.contains { $0.code == mutation.expectedCode },
"\(seed.id) \(mutation.rawValue) expected \(mutation.expectedCode), got \(codes)",
)
#expect(!report.description.isEmpty)
#expect(!report.jsonDescription(title: "Generated Corpus Mutation").isEmpty)
}
}
}
}
private struct DeterministicGenerator {
private var state: UInt64
init(seed: UInt64) {
state = seed
}
mutating func validYAMLDocument(index: Int) -> String {
let service = "service-\(index)-\(nextInt(10000))"
let replicas = nextInt(5) + 1
let enabled = nextInt(2) == 0 ? "true" : "false"
let threshold = "\(nextInt(100)).\(nextInt(10))"
let firstPort = 8000 + nextInt(100)
let secondPort = 9000 + nextInt(100)
return """
---
metadata:
name: \(service)
labels: {app: pureyaml, generated: "true"}
spec:
replicas: \(replicas)
enabled: \(enabled)
threshold: \(threshold)
ports:
- name: http
port: \(firstPort)
- name: metrics
port: \(secondPort)
env:
LOG_LEVEL: "info"
FEATURE_FLAG: "\(enabled)"
"""
}
private mutating func nextInt(_ upperBound: Int) -> Int {
state = state &* 6_364_136_223_846_793_005 &+ 1_442_695_040_888_963_407
return Int((state >> 32) % UInt64(upperBound))
}
}
private enum PropertyMutation: String, CaseIterable {
case missingMappingSpace
case tabIndentation
case missingSequenceSpace
case unterminatedQuotedString
case undefinedAlias
var expectedCode: String {
switch self {
case .missingMappingSpace:
"missingMappingSpace"
case .tabIndentation:
"tabIndentation"
case .missingSequenceSpace:
"missingSequenceSpace"
case .unterminatedQuotedString:
"unterminatedQuotedString"
case .undefinedAlias:
"undefinedAlias"
}
}
func apply(to source: String, seedID: String) -> String {
let sample = source.split(separator: "\n", omittingEmptySubsequences: false)
.prefix(3)
.joined(separator: "\n")
if self == .unterminatedQuotedString {
return """
# mutation-source: \(seedID)
sample: |
\(sample.replacingOccurrences(of: "\n", with: "\n "))
mutation: "unterminated scalar
"""
}
let prefix = switch self {
case .missingMappingSpace:
"mutation:Missing mapping space"
case .tabIndentation:
"\tmutation: tab indentation"
case .missingSequenceSpace:
"-mutation sequence marker"
case .unterminatedQuotedString:
"mutation: \"unterminated scalar"
case .undefinedAlias:
"mutation: *undefinedMutationAlias"
}
return """
# mutation-source: \(seedID)
\(prefix)
sample: |
\(sample.replacingOccurrences(of: "\n", with: "\n "))
"""
}
}
private struct PropertyCorpusSeed {
var id: String
var localPath: String
var tier: String
static func loadAll() throws -> [Self] {
let url = try #require(Bundle.module.url(
forResource: "real-yaml-corpus",
withExtension: "yaml",
subdirectory: "Fixtures",
))
let manifest = try PureYAML.parse(String(contentsOf: url, encoding: .utf8))
let root = try #require(manifest.mapping)
let seedValues = try #require(root.sequence("seeds"))
return try seedValues.map { value in
let mapping = try #require(value.mapping)
return try PropertyCorpusSeed(
id: mapping.requiredString("id"),
localPath: mapping.requiredString("localPath"),
tier: mapping.requiredString("tier"),
)
}
}
func load() throws -> String {
let url = try #require(Bundle.module.url(
forResource: resourceName,
withExtension: resourceExtension,
subdirectory: resourceSubdirectory,
))
return try String(contentsOf: url, encoding: .utf8)
}
private var resourceSubdirectory: String {
let directory = localPath.split(separator: "/").dropLast().joined(separator: "/")
return directory.isEmpty ? "Fixtures" : "Fixtures/\(directory)"
}
private var resourceFileName: String {
String(localPath.split(separator: "/").last ?? "")
}
private var resourceName: String {
guard let dotIndex = resourceFileName.lastIndex(of: ".") else {
return resourceFileName
}
return String(resourceFileName[..<dotIndex])
}
private var resourceExtension: String {
guard let dotIndex = resourceFileName.lastIndex(of: ".") else {
return ""
}
return String(resourceFileName[resourceFileName.index(after: dotIndex)...])
}
}
private extension PureYAML.Model.Mapping {
func requiredString(_ key: String) throws -> String {
guard case let .string(value)? = self[key] else {
throw PropertyCorpusError("missing string key \(key)")
}
return value
}
}
private struct PropertyCorpusError: Error, CustomStringConvertible {
var description: String
init(_ description: String) {
self.description = description
}
}

176
Tests/RealYAMLCorpusTests.swift Normal file
View File

@ -0,0 +1,176 @@
import Foundation
@testable import PureYAML
import Testing
@Suite("Real YAML Corpus")
struct RealYAMLCorpusTests {
@Test("Corpus manifest matches vendored fixture files")
func test_corpusManifestMatchesVendoredFixtureFiles() throws {
let manifest = try RealYAMLCorpusManifest.load()
#expect(manifest.version == 1)
#expect(manifest.seeds.count >= 100)
#expect(manifest.seeds.count(where: { $0.tier == "default" }) == 10)
#expect(manifest.seeds.count(where: { $0.tier == "full" }) >= 90)
#expect(Set(manifest.seeds.map(\.id)).count == manifest.seeds.count)
for seed in manifest.seeds {
let source = try seed.load()
#expect(source.utf8.count == seed.byteCount)
#expect(source.count(where: { $0 == "\n" }) == seed.lineCount)
#expect(seed.commit.count == 40)
#expect(seed.rawURL.contains(seed.commit))
#expect(seed.rawURL.contains(seed.sourcePath))
}
}
@Test("Full real corpus parses validates and round trips when enabled")
func test_fullRealCorpusParsesValidatesAndRoundTripsWhenEnabled() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_RUN_FULL_CORPUS"] == "1" else {
return
}
let manifest = try RealYAMLCorpusManifest.load()
for seed in manifest.seeds {
let source = try seed.load()
let documents: [PureYAML.Stream.Document]
do {
documents = try PureYAML.parseStream(source)
} catch {
recordIssue("\(seed.id): parse failed with \(error)")
continue
}
let issues = PureYAML.Validation.Validator().collect(documents)
let issueDescriptions = issues.issues.map(\.description).joined(separator: "\n")
#expect(issues == PureYAML.Stream.Result(), "\(seed.id): \(issueDescriptions)")
guard seed.expectedRoundTrip == "success" else {
continue
}
let dumped = PureYAML.dump(documents)
do {
let reparsed = try PureYAML.parseStream(dumped)
#expect(reparsed == documents, "\(seed.id): parse/dump/parse changed the document stream")
} catch {
recordIssue("\(seed.id): dumped YAML failed to parse with \(error)")
}
}
}
}
private struct RealYAMLCorpusManifest {
var version: Int
var seeds: [RealYAMLCorpusSeed]
static func load() throws -> Self {
let url = try #require(Bundle.module.url(
forResource: "real-yaml-corpus",
withExtension: "yaml",
subdirectory: "Fixtures",
))
let value = try PureYAML.parse(String(contentsOf: url, encoding: .utf8))
let root = try #require(value.mapping)
let version = try #require(root["version"]?.intValue)
let seedValues = try #require(root.sequence("seeds"))
return try RealYAMLCorpusManifest(
version: version,
seeds: seedValues.map(RealYAMLCorpusSeed.init(value:)),
)
}
}
private struct RealYAMLCorpusSeed {
var id: String
var localPath: String
var category: String
var size: String
var tier: String
var expectedParse: String
var expectedValidation: String
var expectedRoundTrip: String
var byteCount: Int
var lineCount: Int
var repository: String
var commit: String
var sourcePath: String
var rawURL: String
var license: String
init(value: PureYAML.Model.Value) throws {
let mapping = try #require(value.mapping)
id = try mapping.requiredString("id")
localPath = try mapping.requiredString("localPath")
category = try mapping.requiredString("category")
size = try mapping.requiredString("size")
tier = try mapping.requiredString("tier")
expectedParse = try mapping.requiredString("expectedParse")
expectedValidation = try mapping.requiredString("expectedValidation")
expectedRoundTrip = try mapping.requiredString("expectedRoundTrip")
byteCount = try mapping.requiredInt("byteCount")
lineCount = try mapping.requiredInt("lineCount")
repository = try mapping.requiredString("repository")
commit = try mapping.requiredString("commit")
sourcePath = try mapping.requiredString("sourcePath")
rawURL = try mapping.requiredString("rawURL")
license = try mapping.requiredString("license")
}
func load() throws -> String {
let url = try #require(Bundle.module.url(
forResource: resourceName,
withExtension: resourceExtension,
subdirectory: resourceSubdirectory,
))
return try String(contentsOf: url, encoding: .utf8)
}
private var resourceSubdirectory: String {
let directory = localPath.split(separator: "/").dropLast().joined(separator: "/")
return directory.isEmpty ? "Fixtures" : "Fixtures/\(directory)"
}
private var resourceFileName: String {
String(localPath.split(separator: "/").last ?? "")
}
private var resourceName: String {
guard let dotIndex = resourceFileName.lastIndex(of: ".") else {
return resourceFileName
}
return String(resourceFileName[..<dotIndex])
}
private var resourceExtension: String {
guard let dotIndex = resourceFileName.lastIndex(of: ".") else {
return ""
}
return String(resourceFileName[resourceFileName.index(after: dotIndex)...])
}
}
private extension PureYAML.Model.Mapping {
func requiredString(_ key: String) throws -> String {
try #require(self[key]?.stringValue)
}
func requiredInt(_ key: String) throws -> Int {
try #require(self[key]?.intValue)
}
}
private extension PureYAML.Model.Value {
var stringValue: String? {
guard case let .string(value) = self else {
return nil
}
return value
}
var intValue: Int? {
guard case let .int(value) = self else {
return nil
}
return value
}
}

20
Tests/ScannerTests.swift
View File

@ -223,6 +223,26 @@ struct ScannerTests {
expectMarks(value, start: .init(line: 1, column: 7, index: 7), end: .init(line: 1, column: 10, index: 10))
}
@Test("Tracks ASCII and extended UTF-8 marks in one scan")
func test_asciiAndExtendedUTF8Marks() throws {
let tokens = try PureYAML.Parsing.Scanner().scan("ascii: abc\nemoji: 😀\nnext: ok")
guard let emoji = tokens.first(where: {
$0.kind == .scalar(value: "😀", style: .plain)
}) else {
recordIssue("expected emoji scalar")
return
}
guard let next = tokens.first(where: {
$0.kind == .scalar(value: "next", style: .plain)
}) else {
recordIssue("expected next scalar")
return
}
expectMarks(emoji, start: .init(line: 2, column: 8, index: 18), end: .init(line: 2, column: 9, index: 22))
expectMarks(next, start: .init(line: 3, column: 1, index: 23), end: .init(line: 3, column: 5, index: 27))
}
@Test("Reader advances UTF-8 marks across line breaks")
func test_readerAdvancesUTF8MarksAcrossLineBreaks() {
var reader = PureYAML.Parsing.Reader("a\u{00E9}\n\u{03B2}\r\nz")

12
Tests/ValidationReportTests.swift
View File

@ -20,13 +20,15 @@ struct ValidationReportTests {
#expect(report.diagnostics == [
.init(
kind: .parse,
code: "unterminatedQuotedString",
severity: .error,
file: "broken.yaml",
line: 1,
reason: "unterminated quoted string at line 1",
),
])
#expect(!report.isValid)
#expect(report.description == "broken.yaml: error: parse: unterminated quoted string at line 1")
#expect(report.description == "broken.yaml: line 1: error: parse: unterminated quoted string at line 1")
}
@Test("Duplicate keys return document and path diagnostics")
@ -113,7 +115,7 @@ struct ValidationReportTests {
#expect(report.diagnosticCount == 3)
#expect(!report.isValid)
#expect(report.description == [
"broken.yaml: error: parse: unterminated quoted string at line 1",
"broken.yaml: line 1: error: parse: unterminated quoted string at line 1",
"duplicates.yaml: document[0]: $.title: error: validation: Duplicate mapping key 'title'",
"warnings.yaml: document[0]: $.mode: warning: validation: Failed to satisfy: Mode is not legacy",
].joined(separator: "\n"))
@ -292,7 +294,7 @@ No diagnostics.
Status: invalid
```text
broken.yaml: error: parse: unterminated quoted string at line 1
broken.yaml: line 1: error: parse: unterminated quoted string at line 1
```
## duplicates.yaml
@ -328,14 +330,16 @@ private let expectedReportModelValue = PureYAML.Model.Value.mapping(.init([
.init(key: "diagnostics", value: .sequence([
.mapping(.init([
.init(key: "kind", value: .string("parse")),
.init(key: "code", value: .string("unterminatedQuotedString")),
.init(key: "severity", value: .string("error")),
.init(key: "file", value: .string("broken.yaml")),
.init(key: "line", value: .int(1)),
.init(key: "documentIndex", value: .null),
.init(key: "path", value: .null),
.init(key: "reason", value: .string("unterminated quoted string at line 1")),
.init(
key: "description",
value: .string("broken.yaml: error: parse: unterminated quoted string at line 1"),
value: .string("broken.yaml: line 1: error: parse: unterminated quoted string at line 1"),
),
])),
])),

133
docs/CORPUS.md
View File

@ -6,8 +6,121 @@ people actually maintain.
## Current State
The corpus gate is designed but not implemented yet. Implementation is tracked
by:
The checked-in corpus gate is implemented. `Tests/Fixtures/real-yaml-corpus.yaml`
is the machine-readable manifest for 114 pinned public YAML seeds copied under
`Tests/Fixtures/real-yaml/`.
Normal `swift test` verifies the manifest and vendored file metadata without
running the full corpus parse workload. It also runs deterministic generated
properties over small valid YAML documents and malformed mutations derived from
default real-corpus seeds. Run the full checked-in corpus gate explicitly when
hardening parser behavior:
```sh
bash scripts/check-corpus.sh
```
The opt-in gate sets `PUREYAML_RUN_FULL_CORPUS=1` and runs
`RealYAMLCorpusTests`. It stream-parses every seed, validates every document,
and checks parse/dump/parse stability for seeds marked as round-trippable in the
manifest. A successful run writes corpus artifacts under
`.build/pureyaml-artifacts/`.
The normal `parse` and `parseStream` paths now scan tokens lazily and compose
values from parser events as those events are produced. They no longer retain a
full token array or a full intermediate event array. `parseEvents` remains
available as a debugging and compatibility surface that materializes events
explicitly. This is an internal memory-reduction step; the public API still
accepts a complete `String`, so true chunked input parsing remains tracked by
#43.
`RealYAMLCorpusPropertyTests` runs in normal verification and covers:
- 100 deterministic generated valid YAML documents with parse/dump/parse
stability;
- malformed real-seed mutations for missing mapping spaces, tab indentation,
broken sequence markers, broken quoted scalars, and undefined aliases;
- structured diagnostics for every generated malformed mutation.
Run the JSON compatibility gate to prove YAML 1.2 JSON-subset behavior against
Swift's JSON decoder:
```sh
bash scripts/check-json-compatibility.sh
```
The gate covers object, array, and scalar roots; JSON escapes; Unicode escape
sequences including surrogate pairs; exponent numbers; nested values; and
insignificant whitespace. It writes
`.build/pureyaml-artifacts/json/json-compatibility.json`.
Run the Yams comparison gate separately because it intentionally uses an
external dependency in a temporary package under `.build/`:
```sh
bash scripts/check-yams-differential.sh
```
This preserves `Package.swift`'s dependency-free contract while producing
`.build/pureyaml-artifacts/differential/yams-comparison.json`.
The manifest can mark intentional differences. The current checked-in corpus has
one: a comment-only values file where PureYAML returns a single `null` document
and Yams reports zero stream documents. The differential gate accepts only
manifest-declared intentional differences.
Malformed-input diagnostics are compared separately:
```sh
bash scripts/check-yams-diagnostics.sh
```
This gate checks production-shaped bad YAML cases where PureYAML emits
machine-readable diagnostic codes, locations, severities, and descriptions, then
records the corresponding Yams parser result in
`.build/pureyaml-artifacts/diagnostics/yams-diagnostics.json`.
Parsed-document validation is compared separately:
```sh
bash scripts/check-yams-validation.sh
```
This gate checks YAML that both parsers can load but that contains semantic
validation issues such as duplicate mapping keys. PureYAML reports structured
validation paths and reasons; the artifact records Yams' parser result and zero
structured validation issues at
`.build/pureyaml-artifacts/validation/yams-validation.json`.
Throughput comparison is also opt-in and uses the same temporary-package
pattern:
```sh
bash scripts/check-throughput.sh
```
It measures representative checked-in corpus seeds against Yams in release mode
and writes `.build/pureyaml-artifacts/performance/yams-throughput.json`.
PureYAML phase profiling is a separate release-mode gate:
```sh
bash scripts/check-performance-phases.sh
```
It times scanner, token-to-event parsing, event composition, lazy full parsing,
validation, and dumping for the slow representative seeds. It writes
`.build/pureyaml-artifacts/performance/phase-profile.json` so parser
optimization work can target the dominant phase instead of guessing from total
Yams comparison time. The scanner now keeps a UTF-8 cursor for byte-accurate
source marks, which avoids per-character string allocation in ASCII-heavy
advancement while preserving Unicode and CRLF behavior.
Round-trip expectations are tracked separately. A seed can remain in the parse,
validation, and differential gates while skipping parse/dump/parse equality when
the dumper does not yet preserve that document shape.
Further implementation is tracked by:
- #54, the production hardening epic.
- #55, the real-world seed, fuzz, and property-based validation gate.
@ -80,14 +193,20 @@ be committed in `Tests/Fixtures/`.
Fast/default gate:
- at least 25 pinned public real-world YAML seeds,
- small enough for normal verification,
- 114 pinned public real-world YAML seeds copied into `Tests/Fixtures/real-yaml/`,
- normal verification checks the manifest and file metadata only,
- runs on macOS and Linux by default,
- documents Windows and WASI depth if they run a smaller subset.
Full opt-in gate:
- at least 100 pinned public real-world YAML seeds,
- first opt-in gate covers the 114 checked-in seeds with parse, validation, and
round-trip properties,
- default generated properties cover valid document stability and malformed
real-seed mutation diagnostics,
- Yams differential gate compares parse success and stream document counts for
the checked-in corpus,
- target is met with at least 100 pinned public real-world YAML seeds,
- includes at least 20 very large OpenAPI files,
- includes at least 20 non-API config files,
- includes valid and intentionally invalid upstream fixtures where available.
@ -128,7 +247,7 @@ failure should be reduced into a stable checked-in regression fixture.
## Artifacts
Corpus runs should write artifacts under `.build/pureyaml-artifacts/`.
Corpus runs write artifacts under `.build/pureyaml-artifacts/`.
```text
.build/pureyaml-artifacts/
@ -165,7 +284,7 @@ while it exists, then preserve the important case as a committed fixture.
## Acceptance Checklist
- The fast corpus has at least 25 pinned public real-world YAML seeds.
- The checked-in corpus has at least 25 pinned public real-world YAML seeds.
- The full corpus has at least 100 pinned public real-world YAML seeds.
- The manifest records provenance and expected outcomes for every seed.
- The mutation gate is derived from real seeds.

20
docs/RELEASE.md
View File

@ -1,8 +1,8 @@
# Release Process
PureYAML versioning is driven by Git tags. The current release is 0.1.1.
PureYAML versioning is driven by Git tags. The current release is 0.1.2.
## 0.1.1 Release
## 0.1.2 Release
- Swift tools version: 6.1
- Product: `PureYAML`
@ -10,7 +10,7 @@ PureYAML versioning is driven by Git tags. The current release is 0.1.1.
- Source targets: Swift only, no C targets and no generated parser targets
- Hosted CI matrix: macOS build and test, Linux build and test, Windows build
and test, WASM build
- Release notes file: `docs/releases/pureyaml-0-1-1.md`
- Release notes file: `docs/releases/pureyaml-0-1-2.md`
## Local Verification
@ -63,19 +63,19 @@ Only run these commands after the local gates pass and hosted CI passes.
```sh
git status --short --branch
git tag -a 0.1.1 -m "PureYAML 0.1.1"
git push origin main 0.1.1
gh release create 0.1.1 \
git tag -a 0.1.2 -m "PureYAML 0.1.2"
git push origin main 0.1.2
gh release create 0.1.2 \
--repo mihaelamj/PureYAML \
--title "PureYAML 0.1.1" \
--notes-file docs/releases/pureyaml-0-1-1.md
--title "PureYAML 0.1.2" \
--notes-file docs/releases/pureyaml-0-1-2.md
```
After publishing, verify the release exists:
```sh
git ls-remote --tags origin 0.1.1
gh release view 0.1.1 --repo mihaelamj/PureYAML
git ls-remote --tags origin 0.1.2
gh release view 0.1.2 --repo mihaelamj/PureYAML
```
## Do Not Tag If

49
docs/releases/pureyaml-0-1-2.md Normal file
View File

@ -0,0 +1,49 @@
# PureYAML 0.1.2
PureYAML 0.1.2 is a production-hardening release focused on real-world corpus
coverage, structured diagnostics, Yams comparison gates, JSON compatibility, and
lower-memory parser internals.
## Highlights
- Add a 114-seed real-world YAML corpus with opt-in full parse, validation, and
round-trip checks.
- Add structured parser diagnostic codes and validation reports for malformed
production-shaped YAML.
- Add JSON-subset compatibility tests against Swift's JSON decoder.
- Add Yams differential, diagnostic, validation, throughput, and phase-profiling
artifact gates without adding Yams to the public package manifest.
- Route normal parsing through lazy token scanning and streamed event
composition so `parse` and `parseStream` no longer retain a full token array or
full intermediate event array.
- Improve ASCII-heavy scanner advancement with a UTF-8 reader cursor while
preserving byte-accurate Unicode and CRLF source marks.
## Current Production Readiness
The checked corpus agrees with Yams on parse success and stream document counts.
PureYAML now has structured validation that reports duplicate keys with paths,
reasons, and severities, while Yams does not provide equivalent structured
validation output.
The public API still accepts complete `String` input. True chunked file/network
input streaming remains deferred to #43.
## Verification
The release candidate was locally verified with:
- `bash scripts/check-all.sh`
- `bash scripts/check-corpus.sh`
- `bash scripts/check-json-compatibility.sh`
- `bash scripts/check-yams-differential.sh`
- `bash scripts/check-yams-diagnostics.sh`
- `bash scripts/check-yams-validation.sh`
- `bash scripts/check-throughput.sh`
- `bash scripts/check-performance-phases.sh`
## Installation
```swift
.package(url: "https://github.com/mihaelamj/PureYAML.git", .upToNextMinor(from: "0.1.2"))
```

140
scripts/check-corpus.sh Executable file
View File

@ -0,0 +1,140 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts"
PUREYAML_RUN_FULL_CORPUS=1 swift test --filter RealYAMLCorpusTests
mkdir -p "$ARTIFACT_DIR"
python3 - "$ROOT_DIR" "$ARTIFACT_DIR" <<'PY'
import json
import platform
import subprocess
import sys
from collections import Counter
from pathlib import Path
root = Path(sys.argv[1])
artifact_dir = Path(sys.argv[2])
manifest_path = root / "Tests" / "Fixtures" / "real-yaml-corpus.yaml"
def parse_manifest(path):
version = None
description = None
seeds = []
current = None
for raw_line in path.read_text().splitlines():
if raw_line.startswith("version: "):
version = int(raw_line.split(": ", 1)[1])
elif raw_line.startswith("description: "):
description = raw_line.split(": ", 1)[1]
elif raw_line.startswith(" - id: "):
if current is not None:
seeds.append(current)
current = {"id": raw_line.split(": ", 1)[1]}
elif current is not None and raw_line.startswith(" "):
key, value = raw_line.strip().split(": ", 1)
if key in {"byteCount", "lineCount"}:
current[key] = int(value)
else:
current[key] = value
if current is not None:
seeds.append(current)
return {"version": version, "description": description, "seeds": seeds}
def command_output(command):
try:
return subprocess.check_output(
command,
cwd=root,
stderr=subprocess.STDOUT,
text=True,
).strip()
except Exception as error:
return f"unavailable: {error}"
manifest = parse_manifest(manifest_path)
seeds = manifest["seeds"]
categories = Counter(seed["category"] for seed in seeds)
tiers = Counter(seed["tier"] for seed in seeds)
sizes = Counter(seed["size"] for seed in seeds)
summary = {
"seedCount": len(seeds),
"defaultSeeds": tiers["default"],
"fullSeeds": tiers["full"],
"totalBytes": sum(seed["byteCount"] for seed in seeds),
"totalLines": sum(seed["lineCount"] for seed in seeds),
"categories": dict(sorted(categories.items())),
"sizes": dict(sorted(sizes.items())),
}
generated_summary = {
"validDocumentSeed": "0x505559414D4C",
"validDocumentCases": 100,
"mutationSourceTier": "default",
"mutationSourceSeeds": tiers["default"],
"mutationClasses": [
"missingMappingSpace",
"tabIndentation",
"missingSequenceSpace",
"unterminatedQuotedString",
"undefinedAlias",
],
"totalMutations": tiers["default"] * 5,
"status": "passed",
}
(artifact_dir / "real-seed-manifest.json").write_text(
json.dumps(manifest, indent=2, sort_keys=True) + "\n"
)
(artifact_dir / "seeds.json").write_text(
json.dumps(
{
"generated": generated_summary,
"realSeeds": [{"id": seed["id"], "tier": seed["tier"]} for seed in seeds],
},
indent=2,
sort_keys=True,
) + "\n"
)
(artifact_dir / "generated-validation-summary.json").write_text(
json.dumps(generated_summary, indent=2, sort_keys=True) + "\n"
)
def yaml_mapping(mapping, indentation=0):
lines = []
spaces = " " * indentation
for key, value in mapping.items():
if isinstance(value, dict):
lines.append(f"{spaces}{key}:")
lines.extend(yaml_mapping(value, indentation + 2))
elif isinstance(value, list):
lines.append(f"{spaces}{key}:")
for item in value:
lines.append(f"{spaces} - {item}")
else:
lines.append(f"{spaces}{key}: {value}")
return lines
(artifact_dir / "real-seed-summary.yaml").write_text(
"\n".join(yaml_mapping(summary)) + "\n"
)
(artifact_dir / "generated-validation-summary.yaml").write_text(
"\n".join(yaml_mapping(generated_summary)) + "\n"
)
(artifact_dir / "environment.txt").write_text(
"\n".join(
[
f"python: {platform.python_version()}",
f"platform: {platform.platform()}",
f"swift: {command_output(['swift', '--version'])}",
f"git: {command_output(['git', 'rev-parse', 'HEAD'])}",
f"git_status: {command_output(['git', 'status', '--short'])}",
]
) + "\n"
)
print(f"corpus artifacts: {artifact_dir}")
PY

224
scripts/check-json-compatibility.sh Executable file
View File

@ -0,0 +1,224 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
WORK_DIR="$ROOT_DIR/.build/json-compatibility"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts/json"
mkdir -p "$WORK_DIR/Sources/JSONCompatibility" "$ARTIFACT_DIR"
cat > "$WORK_DIR/Package.swift" <<EOF
// swift-tools-version: 6.1
import PackageDescription
let package = Package(
name: "PureYAMLJSONCompatibility",
platforms: [
.macOS(.v13),
],
dependencies: [
.package(path: "$ROOT_DIR"),
],
targets: [
.executableTarget(
name: "JSONCompatibility",
dependencies: [
"PureYAML",
],
),
],
)
EOF
cat > "$WORK_DIR/Sources/JSONCompatibility/main.swift" <<'EOF'
import Foundation
import PureYAML
struct ValidCase {
var name: String
var json: String
}
struct InvalidCase {
var name: String
var json: String
}
let arguments = CommandLine.arguments
guard arguments.count == 2 else {
FileHandle.standardError.write(Data("usage: JSONCompatibility <artifact-json>\n".utf8))
Foundation.exit(64)
}
let validCases = [
ValidCase(
name: "compact object",
json: #"{"openapi":"3.0.0","info":{"title":"JSON API","version":"1"},"paths":{},"enabled":true,"count":3}"#,
),
ValidCase(name: "array root", json: #"[{"a":1},{"b":null},{"items":[true,false]}]"#),
ValidCase(name: "string root", json: #""hello""#),
ValidCase(name: "integer root", json: #"123"#),
ValidCase(name: "boolean root", json: #"true"#),
ValidCase(name: "null root", json: #"null"#),
ValidCase(
name: "string escapes",
json: #"{"quote":"\"","slash":"\\","solidus":"\/","backspace":"\b","formfeed":"\f","newline":"line\nnext","return":"a\rb","tab":"a\tb"}"#,
),
ValidCase(name: "unicode escapes", json: #"{"bmp":"\u263A","surrogate":"\uD834\uDD1E"}"#),
ValidCase(name: "number forms", json: #"{"zero":0,"negative":-3,"fraction":1.25,"small":1e-6,"big":6.02E23}"#),
ValidCase(
name: "insignificant whitespace",
json: """
{
"items" : [
{ "url" : "https://api.example.com/a:b" },
{ "window" : "10:30" }
]
}
""",
),
ValidCase(name: "empty containers", json: #"{"object":{},"array":[]}"#),
]
let invalidCases = [
InvalidCase(name: "missing value", json: #"{"a":}"#),
InvalidCase(name: "double comma", json: #"[1,,2]"#),
InvalidCase(name: "unquoted key", json: #"{a:1}"#),
InvalidCase(name: "comment", json: #"{"a":1 // no comments in JSON"#),
InvalidCase(name: "single quoted string", json: #"{'a':1}"#),
]
let validResults = validCases.map { testCase -> [String: Any] in
do {
let json = try JSONOracle.decode(testCase.json)
let yaml = try JSONOracle(pureYAML: PureYAML.parse(testCase.json))
return [
"name": testCase.name,
"acceptedByJSONOracle": true,
"acceptedByPureYAML": true,
"structurallyEqual": yaml == json,
]
} catch {
return [
"name": testCase.name,
"acceptedByJSONOracle": false,
"acceptedByPureYAML": false,
"structurallyEqual": false,
"error": String(describing: error),
]
}
}
let invalidResults = invalidCases.map { testCase -> [String: Any] in
do {
_ = try JSONOracle.decode(testCase.json)
return [
"name": testCase.name,
"rejectedByJSONOracle": false,
]
} catch {
return [
"name": testCase.name,
"rejectedByJSONOracle": true,
]
}
}
let validFailures = validResults.filter { result in
result["structurallyEqual"] as? Bool != true
}
let invalidFailures = invalidResults.filter { result in
result["rejectedByJSONOracle"] as? Bool != true
}
let report: [String: Any] = [
"summary": [
"validCases": validCases.count,
"validCasesStructurallyEqual": validCases.count - validFailures.count,
"invalidCases": invalidCases.count,
"invalidCasesRejectedByOracle": invalidCases.count - invalidFailures.count,
"failures": validFailures.count + invalidFailures.count,
],
"valid": validResults,
"invalid": invalidResults,
]
let artifactURL = URL(fileURLWithPath: arguments[1])
let data = try JSONSerialization.data(withJSONObject: report, options: [.prettyPrinted, .sortedKeys])
try data.write(to: artifactURL)
print("JSON compatibility valid cases: \(validCases.count)")
print("JSON compatibility structurally equal: \(validCases.count - validFailures.count)")
print("JSON invalid oracle rejections: \(invalidCases.count - invalidFailures.count)")
print("Report: \(artifactURL.path)")
if !validFailures.isEmpty || !invalidFailures.isEmpty {
Foundation.exit(1)
}
enum JSONOracle: Equatable, Decodable {
case null
case bool(Bool)
case int(Int)
case double(Double)
case string(String)
case array([JSONOracle])
case object([String: JSONOracle])
init(from decoder: Decoder) throws {
let container = try decoder.singleValueContainer()
if container.decodeNil() {
self = .null
} else if let value = try? container.decode(Bool.self) {
self = .bool(value)
} else if let value = try? container.decode(Int.self) {
self = .int(value)
} else if let value = try? container.decode(Double.self) {
self = .double(value)
} else if let value = try? container.decode(String.self) {
self = .string(value)
} else if let value = try? container.decode([JSONOracle].self) {
self = .array(value)
} else {
self = .object(try container.decode([String: JSONOracle].self))
}
}
init(pureYAML value: PureYAML.Model.Value) throws {
switch value {
case .null:
self = .null
case let .bool(value):
self = .bool(value)
case let .int(value):
self = .int(value)
case let .double(value):
self = .double(value)
case let .string(value):
self = .string(value)
case let .sequence(values):
self = .array(try values.map(JSONOracle.init(pureYAML:)))
case let .mapping(mapping):
self = .object(try mapping.pairs.reduce(into: [:]) { result, pair in
guard case let .string(key) = pair.keyNode.value else {
throw JSONOracleFailure.nonStringKey
}
result[key] = try JSONOracle(pureYAML: pair.value)
})
}
}
static func decode(_ json: String) throws -> JSONOracle {
try JSONDecoder().decode(JSONOracle.self, from: Data(json.utf8))
}
}
enum JSONOracleFailure: Error {
case nonStringKey
}
EOF
swift run \
--package-path "$WORK_DIR" \
JSONCompatibility \
"$ARTIFACT_DIR/json-compatibility.json"

14
scripts/check-performance-phases.sh Executable file
View File

@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts/performance"
mkdir -p "$ARTIFACT_DIR"
PUREYAML_RUN_PHASE_PROFILER=1 \
PUREYAML_PHASE_PROFILER_ARTIFACT="$ARTIFACT_DIR/phase-profile.json" \
swift test \
-c release \
--package-path "$ROOT_DIR" \
--filter PerformancePhaseTests

182
scripts/check-throughput.sh Executable file
View File

@ -0,0 +1,182 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
WORK_DIR="$ROOT_DIR/.build/yams-throughput"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts/performance"
mkdir -p "$WORK_DIR/Sources/YamsThroughput" "$ARTIFACT_DIR"
cat > "$WORK_DIR/Package.swift" <<EOF
// swift-tools-version: 6.1
import PackageDescription
let package = Package(
name: "PureYAMLYamsThroughput",
platforms: [
.macOS(.v13),
],
dependencies: [
.package(path: "$ROOT_DIR"),
.package(url: "https://github.com/jpsim/Yams.git", from: "5.0.0"),
],
targets: [
.executableTarget(
name: "YamsThroughput",
dependencies: [
"PureYAML",
.product(name: "Yams", package: "Yams"),
],
),
],
)
EOF
cat > "$WORK_DIR/Sources/YamsThroughput/main.swift" <<'EOF'
import Foundation
import PureYAML
import Yams
struct Seed {
var id: String
var localPath: String
var iterations: Int
}
struct Measurement {
var seed: Seed
var bytes: Int
var pureSeconds: TimeInterval
var yamsSeconds: TimeInterval
var pureDocuments: Int
var yamsDocuments: Int
var pureMegabytesPerSecond: Double {
Double(bytes * seed.iterations) / pureSeconds / 1_000_000
}
var yamsMegabytesPerSecond: Double {
Double(bytes * seed.iterations) / yamsSeconds / 1_000_000
}
var ratio: Double {
pureMegabytesPerSecond / yamsMegabytesPerSecond
}
var object: [String: Any] {
[
"id": seed.id,
"localPath": seed.localPath,
"bytes": bytes,
"iterations": seed.iterations,
"pureyaml": [
"seconds": pureSeconds,
"megabytesPerSecond": pureMegabytesPerSecond,
"documents": pureDocuments,
],
"yams": [
"seconds": yamsSeconds,
"megabytesPerSecond": yamsMegabytesPerSecond,
"documents": yamsDocuments,
],
"pureyamlToYamsRatio": ratio,
]
}
}
let arguments = CommandLine.arguments
guard arguments.count == 3 else {
FileHandle.standardError.write(Data("usage: YamsThroughput <repo-root> <artifact-json>\n".utf8))
Foundation.exit(64)
}
let rootURL = URL(fileURLWithPath: arguments[1], isDirectory: true)
let artifactURL = URL(fileURLWithPath: arguments[2])
let fixtureURL = rootURL
.appendingPathComponent("Tests")
.appendingPathComponent("Fixtures")
.appendingPathComponent("real-yaml")
let seeds = [
Seed(id: "petstore", localPath: "openapi-petstore.yaml", iterations: 100),
Seed(id: "prometheus", localPath: "prometheus-conf-good.yml", iterations: 60),
Seed(id: "cert-manager-crd", localPath: "cert-manager-certificate-crd.yaml", iterations: 20),
Seed(id: "stripe-yaml", localPath: "api-guru-stripe-openapi.yaml", iterations: 2),
Seed(id: "zoom-yaml", localPath: "api-guru-zoom-openapi.yaml", iterations: 2),
]
let measurements = try seeds.map { seed in
let url = fixtureURL.appendingPathComponent(seed.localPath)
let yaml = try String(contentsOf: url, encoding: .utf8)
_ = try PureYAML.parseStream(yaml)
_ = try Array(Yams.load_all(yaml: yaml))
let pure = try timed {
var documents = 0
for _ in 0..<seed.iterations {
documents += try PureYAML.parseStream(yaml).count
}
return documents
}
let yams = try timed {
var documents = 0
for _ in 0..<seed.iterations {
documents += try Array(Yams.load_all(yaml: yaml)).count
}
return documents
}
return Measurement(
seed: seed,
bytes: yaml.utf8.count,
pureSeconds: pure.seconds,
yamsSeconds: yams.seconds,
pureDocuments: pure.value,
yamsDocuments: yams.value,
)
}
let report: [String: Any] = [
"summary": [
"seeds": measurements.count,
"pureyamlFasterOrEqual": measurements.filter { $0.ratio >= 1 }.count,
"pureyamlSlower": measurements.filter { $0.ratio < 1 }.count,
"minimumRatio": measurements.map(\.ratio).min() ?? 0,
],
"measurements": measurements.map(\.object),
]
let data = try JSONSerialization.data(withJSONObject: report, options: [.prettyPrinted, .sortedKeys])
try data.write(to: artifactURL)
print("Yams throughput corpus seeds: \(measurements.count)")
for measurement in measurements {
print([
measurement.seed.id,
"bytes=\(measurement.bytes)",
"iter=\(measurement.seed.iterations)",
"pure=\(format(measurement.pureMegabytesPerSecond))MB/s",
"yams=\(format(measurement.yamsMegabytesPerSecond))MB/s",
"ratio=\(format(measurement.ratio))",
].joined(separator: " "))
}
print("Report: \(artifactURL.path)")
func timed(_ body: () throws -> Int) rethrows -> (seconds: TimeInterval, value: Int) {
let start = DispatchTime.now().uptimeNanoseconds
let value = try body()
let end = DispatchTime.now().uptimeNanoseconds
return (Double(end - start) / 1_000_000_000, value)
}
func format(_ value: Double) -> String {
String(format: "%.2f", value)
}
EOF
swift run \
-c release \
--package-path "$WORK_DIR" \
YamsThroughput \
"$ROOT_DIR" \
"$ARTIFACT_DIR/yams-throughput.json"

203
scripts/check-yams-diagnostics.sh Executable file
View File

@ -0,0 +1,203 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
WORK_DIR="$ROOT_DIR/.build/yams-diagnostics"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts/diagnostics"
mkdir -p "$WORK_DIR/Sources/YamsDiagnostics" "$ARTIFACT_DIR"
cat > "$WORK_DIR/Package.swift" <<EOF
// swift-tools-version: 6.1
import PackageDescription
let package = Package(
name: "PureYAMLYamsDiagnostics",
platforms: [
.macOS(.v13),
],
dependencies: [
.package(path: "$ROOT_DIR"),
.package(url: "https://github.com/jpsim/Yams.git", from: "5.0.0"),
],
targets: [
.executableTarget(
name: "YamsDiagnostics",
dependencies: [
"PureYAML",
.product(name: "Yams", package: "Yams"),
],
),
],
)
EOF
cat > "$WORK_DIR/Sources/YamsDiagnostics/main.swift" <<'EOF'
import Foundation
import PureYAML
import Yams
struct Case {
var id: String
var yaml: String
var expectedCodes: [String]
}
struct Comparison {
var testCase: Case
var pureDiagnostics: [PureYAML.Validation.Diagnostic]
var yamsSuccess: Bool
var yamsDocumentCount: Int?
var yamsError: String?
var hasExpectedCodes: Bool {
let codes = Set(pureDiagnostics.compactMap(\.code))
return testCase.expectedCodes.allSatisfy { codes.contains($0) }
}
var object: [String: Any] {
[
"id": testCase.id,
"expectedCodes": testCase.expectedCodes,
"pureyaml": [
"diagnostics": pureDiagnostics.map(diagnosticObject),
"diagnosticCount": pureDiagnostics.count,
"errorCount": pureDiagnostics.filter { $0.severity == .error }.count,
"warningCount": pureDiagnostics.filter { $0.severity == .warning }.count,
"hasExpectedCodes": hasExpectedCodes,
],
"yams": [
"success": yamsSuccess,
"documentCount": yamsDocumentCount as Any,
"error": yamsError as Any,
"structuredDiagnostics": false,
],
]
}
}
let arguments = CommandLine.arguments
guard arguments.count == 2 else {
FileHandle.standardError.write(Data("usage: YamsDiagnostics <artifact-json>\n".utf8))
Foundation.exit(64)
}
let artifactURL = URL(fileURLWithPath: arguments[1])
let cases = [
Case(
id: "missing-mapping-space",
yaml: "apiVersion:v1\nkind: Pod\n",
expectedCodes: ["missingMappingSpace"],
),
Case(
id: "tab-indentation",
yaml: "root:\n\tchild: value\n",
expectedCodes: ["tabIndentation"],
),
Case(
id: "missing-sequence-space",
yaml: "-bad\n- ok\n",
expectedCodes: ["missingSequenceSpace"],
),
Case(
id: "unterminated-quoted-string",
yaml: "title: \"open\nsummary: still scanned\n",
expectedCodes: ["unterminatedQuotedString"],
),
Case(
id: "production-shaped-batch",
yaml: [
"apiVersion:v1",
"\tmetadata:",
"-bad",
"name: demo ",
"image:nginx",
"\t- item",
"-item",
"version:1",
"ports: [80]\t",
"owner:team",
"title: \"open",
].joined(separator: "\n"),
expectedCodes: [
"missingMappingSpace",
"tabIndentation",
"missingSequenceSpace",
"trailingWhitespace",
"unterminatedQuotedString",
],
),
]
let comparisons = cases.map { testCase in
let report = PureYAML.diagnosticValidationReport(testCase.yaml, file: "\(testCase.id).yaml")
let yams = parseWithYams(testCase.yaml)
return Comparison(
testCase: testCase,
pureDiagnostics: report.diagnostics,
yamsSuccess: yams.success,
yamsDocumentCount: yams.documentCount,
yamsError: yams.error,
)
}
let failures = comparisons.filter { $0.pureDiagnostics.isEmpty || !$0.hasExpectedCodes }
let report: [String: Any] = [
"summary": [
"cases": comparisons.count,
"pureyamlStructuredCases": comparisons.filter { !$0.pureDiagnostics.isEmpty }.count,
"pureyamlCasesWithExpectedCodes": comparisons.filter(\.hasExpectedCodes).count,
"yamsStructuredCases": 0,
"failures": failures.count,
],
"comparisons": comparisons.map(\.object),
]
let data = try JSONSerialization.data(withJSONObject: report, options: [.prettyPrinted, .sortedKeys])
try data.write(to: artifactURL)
print("Yams diagnostic comparison cases: \(comparisons.count)")
print("PureYAML structured cases: \(comparisons.filter { !$0.pureDiagnostics.isEmpty }.count)")
print("PureYAML cases with expected codes: \(comparisons.filter(\.hasExpectedCodes).count)")
print("Yams structured cases: 0")
print("Report: \(artifactURL.path)")
if !failures.isEmpty {
for comparison in failures {
print("DIAGNOSTIC GAP \(comparison.testCase.id)")
print(" expected codes: \(comparison.testCase.expectedCodes.joined(separator: ", "))")
print(" actual codes: \(comparison.pureDiagnostics.compactMap(\.code).joined(separator: ", "))")
}
Foundation.exit(1)
}
func parseWithYams(_ yaml: String) -> (success: Bool, documentCount: Int?, error: String?) {
do {
let documents = try Yams.load_all(yaml: yaml)
return (true, Array(documents).count, nil)
} catch {
return (false, nil, String(describing: error))
}
}
func diagnosticObject(_ diagnostic: PureYAML.Validation.Diagnostic) -> [String: Any] {
[
"kind": diagnostic.kind.description,
"code": diagnostic.code as Any,
"severity": diagnostic.severity.description,
"file": diagnostic.file as Any,
"line": diagnostic.line as Any,
"column": diagnostic.column as Any,
"documentIndex": diagnostic.documentIndex as Any,
"path": diagnostic.path?.description as Any,
"reason": diagnostic.reason,
"description": diagnostic.description,
]
}
EOF
swift run \
--package-path "$WORK_DIR" \
YamsDiagnostics \
"$ARTIFACT_DIR/yams-diagnostics.json"

221
scripts/check-yams-differential.sh Executable file
View File

@ -0,0 +1,221 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
WORK_DIR="$ROOT_DIR/.build/yams-differential"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts/differential"
mkdir -p "$WORK_DIR/Sources/YamsDifferential" "$ARTIFACT_DIR"
cat > "$WORK_DIR/Package.swift" <<EOF
// swift-tools-version: 6.1
import PackageDescription
let package = Package(
name: "PureYAMLYamsDifferential",
platforms: [
.macOS(.v13),
],
dependencies: [
.package(path: "$ROOT_DIR"),
.package(url: "https://github.com/jpsim/Yams.git", from: "5.0.0"),
],
targets: [
.executableTarget(
name: "YamsDifferential",
dependencies: [
"PureYAML",
.product(name: "Yams", package: "Yams"),
],
),
],
)
EOF
cat > "$WORK_DIR/Sources/YamsDifferential/main.swift" <<'EOF'
import Foundation
import PureYAML
import Yams
struct Seed {
var id: String
var localPath: String
var category: String
var tier: String
var repository: String
var commit: String
var sourcePath: String
var expectedDifferential: String
}
struct Comparison {
var seed: Seed
var pureSuccess: Bool
var pureDocumentCount: Int?
var pureError: String?
var yamsSuccess: Bool
var yamsDocumentCount: Int?
var yamsError: String?
var agrees: Bool {
if seed.expectedDifferential == "yamsEmptyStreamZeroDocuments" {
return pureSuccess && yamsSuccess && pureDocumentCount == 1 && yamsDocumentCount == 0
}
return pureSuccess == yamsSuccess && pureDocumentCount == yamsDocumentCount
}
var object: [String: Any] {
[
"id": seed.id,
"localPath": seed.localPath,
"category": seed.category,
"tier": seed.tier,
"repository": seed.repository,
"commit": seed.commit,
"sourcePath": seed.sourcePath,
"expectedDifferential": seed.expectedDifferential,
"agrees": agrees,
"pureyaml": [
"success": pureSuccess,
"documentCount": pureDocumentCount as Any,
"error": pureError as Any,
],
"yams": [
"success": yamsSuccess,
"documentCount": yamsDocumentCount as Any,
"error": yamsError as Any,
],
]
}
}
let arguments = CommandLine.arguments
guard arguments.count == 3 else {
FileHandle.standardError.write(Data("usage: YamsDifferential <repo-root> <artifact-json>\n".utf8))
Foundation.exit(64)
}
let rootURL = URL(fileURLWithPath: arguments[1], isDirectory: true)
let artifactURL = URL(fileURLWithPath: arguments[2])
let manifestURL = rootURL
.appendingPathComponent("Tests")
.appendingPathComponent("Fixtures")
.appendingPathComponent("real-yaml-corpus.yaml")
let manifest = try PureYAML.parse(String(contentsOf: manifestURL, encoding: .utf8))
let seeds = try loadSeeds(from: manifest)
var comparisons: [Comparison] = []
for seed in seeds {
let sourceURL = rootURL
.appendingPathComponent("Tests")
.appendingPathComponent("Fixtures")
.appendingPathComponent(seed.localPath)
let yaml = try String(contentsOf: sourceURL, encoding: .utf8)
let pure = parseWithPureYAML(yaml)
let yams = parseWithYams(yaml)
comparisons.append(Comparison(
seed: seed,
pureSuccess: pure.success,
pureDocumentCount: pure.documentCount,
pureError: pure.error,
yamsSuccess: yams.success,
yamsDocumentCount: yams.documentCount,
yamsError: yams.error,
))
}
let disagreements = comparisons.filter { !$0.agrees }
let report: [String: Any] = [
"summary": [
"seeds": comparisons.count,
"agreements": comparisons.count - disagreements.count,
"disagreements": disagreements.count,
],
"comparisons": comparisons.map(\.object),
]
let data = try JSONSerialization.data(withJSONObject: report, options: [.prettyPrinted, .sortedKeys])
try data.write(to: artifactURL)
print("Yams differential corpus seeds: \(comparisons.count)")
print("Agreements: \(comparisons.count - disagreements.count)")
print("Disagreements: \(disagreements.count)")
print("Report: \(artifactURL.path)")
if !disagreements.isEmpty {
for comparison in disagreements {
print("DISAGREE \(comparison.seed.id)")
print(" PureYAML: \(comparison.pureSuccess) documents=\(comparison.pureDocumentCount.map(String.init) ?? "nil") error=\(comparison.pureError ?? "nil")")
print(" Yams: \(comparison.yamsSuccess) documents=\(comparison.yamsDocumentCount.map(String.init) ?? "nil") error=\(comparison.yamsError ?? "nil")")
}
Foundation.exit(1)
}
func parseWithPureYAML(_ yaml: String) -> (success: Bool, documentCount: Int?, error: String?) {
do {
let documents = try PureYAML.parseStream(yaml)
return (true, documents.count, nil)
} catch {
return (false, nil, String(describing: error))
}
}
func parseWithYams(_ yaml: String) -> (success: Bool, documentCount: Int?, error: String?) {
do {
let documents = try Yams.load_all(yaml: yaml)
return (true, Array(documents).count, nil)
} catch {
return (false, nil, String(describing: error))
}
}
func loadSeeds(from value: PureYAML.Model.Value) throws -> [Seed] {
guard case let .mapping(root) = value,
case let .sequence(seedValues)? = root["seeds"]
else {
throw Failure("real-yaml-corpus.yaml must contain a seeds sequence")
}
return try seedValues.map { value in
guard case let .mapping(mapping) = value else {
throw Failure("seed entry must be a mapping")
}
return Seed(
id: try mapping.requiredString("id"),
localPath: try mapping.requiredString("localPath"),
category: try mapping.requiredString("category"),
tier: try mapping.requiredString("tier"),
repository: try mapping.requiredString("repository"),
commit: try mapping.requiredString("commit"),
sourcePath: try mapping.requiredString("sourcePath"),
expectedDifferential: try mapping.requiredString("expectedDifferential"),
)
}
}
struct Failure: Error, CustomStringConvertible {
var description: String
init(_ description: String) {
self.description = description
}
}
extension PureYAML.Model.Mapping {
func requiredString(_ key: String) throws -> String {
guard case let .string(value)? = self[key] else {
throw Failure("seed entry missing string key \(key)")
}
return value
}
}
EOF
swift run \
--package-path "$WORK_DIR" \
YamsDifferential \
"$ROOT_DIR" \
"$ARTIFACT_DIR/yams-comparison.json"

202
scripts/check-yams-validation.sh Executable file
View File

@ -0,0 +1,202 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
WORK_DIR="$ROOT_DIR/.build/yams-validation"
ARTIFACT_DIR="$ROOT_DIR/.build/pureyaml-artifacts/validation"
mkdir -p "$WORK_DIR/Sources/YamsValidation" "$ARTIFACT_DIR"
cat > "$WORK_DIR/Package.swift" <<EOF
// swift-tools-version: 6.1
import PackageDescription
let package = Package(
name: "PureYAMLYamsValidation",
platforms: [
.macOS(.v13),
],
dependencies: [
.package(path: "$ROOT_DIR"),
.package(url: "https://github.com/jpsim/Yams.git", from: "5.0.0"),
],
targets: [
.executableTarget(
name: "YamsValidation",
dependencies: [
"PureYAML",
.product(name: "Yams", package: "Yams"),
],
),
],
)
EOF
cat > "$WORK_DIR/Sources/YamsValidation/main.swift" <<'EOF'
import Foundation
import PureYAML
import Yams
struct Case {
var id: String
var yaml: String
var expectedIssueCount: Int
}
struct Comparison {
var testCase: Case
var pureParseSuccess: Bool
var pureIssues: [PureYAML.Validation.Issue]
var pureError: String?
var yamsParseSuccess: Bool
var yamsDocumentCount: Int?
var yamsError: String?
var provesValidationAdvantage: Bool {
pureParseSuccess
&& pureIssues.count == testCase.expectedIssueCount
&& yamsParseSuccess
}
var object: [String: Any] {
[
"id": testCase.id,
"expectedIssueCount": testCase.expectedIssueCount,
"provesValidationAdvantage": provesValidationAdvantage,
"pureyaml": [
"parseSuccess": pureParseSuccess,
"validationIssueCount": pureIssues.count,
"validationIssues": pureIssues.map(issueObject),
"error": pureError as Any,
],
"yams": [
"parseSuccess": yamsParseSuccess,
"documentCount": yamsDocumentCount as Any,
"error": yamsError as Any,
"structuredValidationIssues": 0,
],
]
}
}
let arguments = CommandLine.arguments
guard arguments.count == 2 else {
FileHandle.standardError.write(Data("usage: YamsValidation <artifact-json>\n".utf8))
Foundation.exit(64)
}
let cases = [
Case(
id: "root-duplicate-key",
yaml: """
title: First
title: Second
""",
expectedIssueCount: 1,
),
Case(
id: "nested-block-and-flow-duplicates",
yaml: """
root:
title: First
title: Second
children:
- name: One
name: Two
flow: {id: one, id: two}
""",
expectedIssueCount: 3,
),
Case(
id: "openapi-path-duplicate",
yaml: """
openapi: "3.1.0"
paths:
/users:
get:
operationId: listUsers
/users:
post:
operationId: createUser
""",
expectedIssueCount: 1,
),
]
let comparisons = cases.map { testCase in
let pure = validateWithPureYAML(testCase.yaml)
let yams = parseWithYams(testCase.yaml)
return Comparison(
testCase: testCase,
pureParseSuccess: pure.parseSuccess,
pureIssues: pure.issues,
pureError: pure.error,
yamsParseSuccess: yams.success,
yamsDocumentCount: yams.documentCount,
yamsError: yams.error,
)
}
let failures = comparisons.filter { !$0.provesValidationAdvantage }
let report: [String: Any] = [
"summary": [
"cases": comparisons.count,
"pureyamlStructuredValidationCases": comparisons.filter { !$0.pureIssues.isEmpty }.count,
"yamsStructuredValidationCases": 0,
"failures": failures.count,
],
"comparisons": comparisons.map(\.object),
]
let artifactURL = URL(fileURLWithPath: arguments[1])
let data = try JSONSerialization.data(withJSONObject: report, options: [.prettyPrinted, .sortedKeys])
try data.write(to: artifactURL)
print("Yams validation comparison cases: \(comparisons.count)")
print("PureYAML structured validation cases: \(comparisons.filter { !$0.pureIssues.isEmpty }.count)")
print("Yams structured validation cases: 0")
print("Report: \(artifactURL.path)")
if !failures.isEmpty {
for comparison in failures {
print("VALIDATION GAP \(comparison.testCase.id)")
print(" PureYAML parse=\(comparison.pureParseSuccess) issues=\(comparison.pureIssues.count) error=\(comparison.pureError ?? "nil")")
print(" Yams parse=\(comparison.yamsParseSuccess) docs=\(comparison.yamsDocumentCount.map(String.init) ?? "nil") error=\(comparison.yamsError ?? "nil")")
}
Foundation.exit(1)
}
func validateWithPureYAML(_ yaml: String) -> (parseSuccess: Bool, issues: [PureYAML.Validation.Issue], error: String?) {
do {
let documents = try PureYAML.parseStream(yaml)
let result = PureYAML.Validation.Validator().collect(documents)
return (true, result.issues.map(\.issue), nil)
} catch {
return (false, [], String(describing: error))
}
}
func parseWithYams(_ yaml: String) -> (success: Bool, documentCount: Int?, error: String?) {
do {
let documents = try Yams.load_all(yaml: yaml)
return (true, Array(documents).count, nil)
} catch {
return (false, nil, String(describing: error))
}
}
func issueObject(_ issue: PureYAML.Validation.Issue) -> [String: Any] {
[
"severity": issue.severity.description,
"reason": issue.reason,
"path": issue.path.description,
"description": issue.description,
]
}
EOF
swift run \
--package-path "$WORK_DIR" \
YamsValidation \
"$ARTIFACT_DIR/yams-validation.json"