feat: scaffold PureYAML streaming incubator

Add a companion package with chunked UTF-8 input, document-token streaming, PureYAML-backed document callbacks, corpus parity tests, docs, and CI.

.github/workflows/ci.yml

@@ -0,0 +1,15 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/checkout@v6
+      - name: Build and test
+        run: bash scripts/check.sh

.gitignore

@@ -0,0 +1,3 @@
+.build/
+.swiftpm/
+.DS_Store

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mihaela Mj
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Package.resolved

@@ -0,0 +1,15 @@
+{
+  "originHash" : "b55dccae149bbe8bc8399ad2f239e7d96da2f76dc3c9a7c34f27f944b0ca2057",
+  "pins" : [
+    {
+      "identity" : "pureyaml",
+      "kind" : "remoteSourceControl",
+      "location" : "https://github.com/mihaelamj/PureYAML.git",
+      "state" : {
+        "branch" : "main",
+        "revision" : "57c48cb4e14e9934770f4ce167d0d0acf028d011"
+      }
+    }
+  ],
+  "version" : 3
+}

Package.swift

@@ -0,0 +1,29 @@
+// swift-tools-version: 6.1
+
+import PackageDescription
+
+let package = Package(
+    name: "PureYAMLStreaming",
+    platforms: [
+        .macOS(.v13),
+    ],
+    products: [
+        .library(
+            name: "PureYAMLStreaming",
+            targets: ["PureYAMLStreaming"],
+        ),
+    ],
+    dependencies: [
+        .package(url: "https://github.com/mihaelamj/PureYAML.git", branch: "main"),
+    ],
+    targets: [
+        .target(
+            name: "PureYAMLStreaming",
+            dependencies: ["PureYAML"],
+        ),
+        .testTarget(
+            name: "PureYAMLStreamingTests",
+            dependencies: ["PureYAMLStreaming"],
+        ),
+    ],
+)

README.md

@@ -0,0 +1,50 @@
+# PureYAMLStreaming
+
+Experimental full-streaming incubator for PureYAML.
+
+This package is intentionally separate from `PureYAML` while the chunked input,
+document streaming, and eventual event streaming APIs settle. It depends on
+PureYAML for the stable model and parser semantics, then adds streaming input
+adapters around that stable core.
+
+## Status
+
+Prototype.
+
+The current implementation streams YAML input in chunks, detects
+document-boundary candidates, and emits one parsed `PureYAML.Stream.Document` at
+a time. That gives bounded memory for multi-document streams where each document
+fits in memory.
+
+It is not yet a token-level YAML scanner. A future milestone will replace the
+document-boundary scanner with a resumable token scanner that can feed
+event-level consumers directly.
+
+## Memory Contract
+
+- `parseDocuments` is bounded by the largest current document plus parser state.
+- `collectDocuments` is a convenience helper and retains all returned documents.
+- Anchors, aliases, merge keys, and `PureYAML.Model.Value` construction can
+  require retaining semantic state until the current document is complete.
+- Future event-level APIs should be lower memory than document-level APIs, but
+  they still cannot promise constant memory for every YAML feature.
+
+## Example
+
+```swift
+import Foundation
+import PureYAMLStreaming
+
+let url = URL(fileURLWithPath: "stream.yaml")
+let parser = PureYAMLStreaming.Parser()
+
+try parser.parseDocuments(from: url) { document in
+    print("document", document.index, document.value)
+}
+```
+
+## Relationship To PureYAML
+
+`PureYAML` remains the dependency-free core package. This package is the
+incubator for file, byte, and async streaming APIs. Stable pieces can move back
+into PureYAML later once the API and memory behavior are proven.

Sources/PureYAMLStreaming/ChunkedUTF8Reader.swift

@@ -0,0 +1,149 @@
+import Foundation
+
+extension PureYAMLStreaming {
+    /// Bounded UTF-8 chunk reader with byte-accurate mark tracking.
+    public struct ChunkedUTF8Reader: Sendable {
+        private var source: Source
+        private let chunkSize: Int
+        private var pendingBytes: [UInt8] = []
+        private var markStorage = StreamingMark.start
+        private var previousWasCarriageReturn = false
+
+        public init(fileURL: URL, chunkSize: Int = 64 * 1024) throws {
+            guard chunkSize > 0 else {
+                throw StreamingError.invalidChunkSize(chunkSize)
+            }
+            self.source = .file(try FileHandle(forReadingFrom: fileURL))
+            self.chunkSize = chunkSize
+        }
+
+        public init(data: Data, chunkSize: Int = 64 * 1024) throws {
+            guard chunkSize > 0 else {
+                throw StreamingError.invalidChunkSize(chunkSize)
+            }
+            self.source = .data(data, offset: 0)
+            self.chunkSize = chunkSize
+        }
+
+        public var mark: StreamingMark {
+            markStorage
+        }
+
+        /// Reads the next valid UTF-8 chunk. A multibyte scalar split at a chunk
+        /// boundary is carried to the next read.
+        public mutating func readChunk() throws -> String? {
+            while true {
+                let bytes = try readRawBytes()
+                if bytes.isEmpty {
+                    guard !pendingBytes.isEmpty else {
+                        return nil
+                    }
+                    return try decodePendingAtEnd()
+                }
+
+                let combined = pendingBytes + bytes
+                let splitIndex = validPrefixLength(in: combined)
+                let prefix = Array(combined.prefix(splitIndex))
+                pendingBytes = Array(combined.dropFirst(splitIndex))
+
+                guard pendingBytes.count < 4 else {
+                    throw StreamingError.invalidUTF8(
+                        line: markStorage.line,
+                        column: markStorage.column,
+                        index: markStorage.index,
+                    )
+                }
+
+                guard !prefix.isEmpty else {
+                    continue
+                }
+
+                guard let chunk = String(bytes: prefix, encoding: .utf8) else {
+                    throw StreamingError.invalidUTF8(
+                        line: markStorage.line,
+                        column: markStorage.column,
+                        index: markStorage.index,
+                    )
+                }
+                advanceMark(over: chunk)
+                return chunk
+            }
+        }
+
+        private mutating func readRawBytes() throws -> [UInt8] {
+            switch source {
+            case let .file(handle):
+                let data = try handle.read(upToCount: chunkSize) ?? Data()
+                return Array(data)
+            case let .data(data, offset):
+                guard offset < data.count else {
+                    return []
+                }
+                let end = min(offset + chunkSize, data.count)
+                source = .data(data, offset: end)
+                return Array(data[offset ..< end])
+            }
+        }
+
+        private mutating func decodePendingAtEnd() throws -> String? {
+            guard let chunk = String(bytes: pendingBytes, encoding: .utf8) else {
+                throw StreamingError.invalidUTF8(
+                    line: markStorage.line,
+                    column: markStorage.column,
+                    index: markStorage.index,
+                )
+            }
+            pendingBytes.removeAll(keepingCapacity: true)
+            advanceMark(over: chunk)
+            return chunk.isEmpty ? nil : chunk
+        }
+
+        private func validPrefixLength(in bytes: [UInt8]) -> Int {
+            guard !bytes.isEmpty else {
+                return 0
+            }
+
+            var index = bytes.count
+            let lowerBound = max(0, bytes.count - 4)
+            while index >= lowerBound {
+                if String(bytes: bytes.prefix(index), encoding: .utf8) != nil {
+                    return index
+                }
+                if index == 0 {
+                    break
+                }
+                index -= 1
+            }
+            return 0
+        }
+
+        private mutating func advanceMark(over chunk: String) {
+            for scalar in chunk.unicodeScalars {
+                markStorage.index += scalar.utf8.count
+                switch scalar.value {
+                case 13:
+                    markStorage.line += 1
+                    markStorage.column = 1
+                    previousWasCarriageReturn = true
+                case 10:
+                    if previousWasCarriageReturn {
+                        previousWasCarriageReturn = false
+                    } else {
+                        markStorage.line += 1
+                        markStorage.column = 1
+                    }
+                default:
+                    previousWasCarriageReturn = false
+                    markStorage.column += 1
+                }
+            }
+        }
+    }
+}
+
+extension PureYAMLStreaming.ChunkedUTF8Reader {
+    private enum Source: @unchecked Sendable {
+        case file(FileHandle)
+        case data(Data, offset: Int)
+    }
+}

Sources/PureYAMLStreaming/Parser.swift

@@ -0,0 +1,91 @@
+import Foundation
+import PureYAML
+
+extension PureYAMLStreaming {
+    /// Chunked YAML stream parser that emits complete documents as they finish.
+    public struct Parser: Sendable {
+        public var chunkSize: Int
+        public var pureParser: PureYAML.Parsing.Parser
+
+        public init(
+            chunkSize: Int = 64 * 1024,
+            pureParser: PureYAML.Parsing.Parser = .init(),
+        ) {
+            self.chunkSize = chunkSize
+            self.pureParser = pureParser
+        }
+
+        /// Parses documents from a file and calls `body` once per completed document.
+        ///
+        /// This API retains only the current source document, parser state, and any
+        /// values PureYAML must build for that document. It does not retain prior
+        /// documents unless the caller does.
+        public func parseDocuments(
+            from url: URL,
+            _ body: (PureYAML.Stream.Document) throws -> Void,
+        ) throws {
+            var reader = try ChunkedUTF8Reader(fileURL: url, chunkSize: chunkSize)
+            try parseDocuments(from: &reader, body)
+        }
+
+        /// Parses documents from UTF-8 data and calls `body` once per completed document.
+        public func parseDocuments(
+            data: Data,
+            _ body: (PureYAML.Stream.Document) throws -> Void,
+        ) throws {
+            var reader = try ChunkedUTF8Reader(data: data, chunkSize: chunkSize)
+            try parseDocuments(from: &reader, body)
+        }
+
+        /// Convenience API that collects every streamed document.
+        ///
+        /// This forfeits the streaming memory benefit by retaining all documents.
+        public func collectDocuments(from url: URL) throws -> [PureYAML.Stream.Document] {
+            var documents: [PureYAML.Stream.Document] = []
+            try parseDocuments(from: url) { document in
+                documents.append(document)
+            }
+            return documents
+        }
+
+        /// Convenience API that collects every streamed document from data.
+        public func collectDocuments(data: Data) throws -> [PureYAML.Stream.Document] {
+            var documents: [PureYAML.Stream.Document] = []
+            try parseDocuments(data: data) { document in
+                documents.append(document)
+            }
+            return documents
+        }
+
+        private func parseDocuments(
+            from reader: inout ChunkedUTF8Reader,
+            _ body: (PureYAML.Stream.Document) throws -> Void,
+        ) throws {
+            var scanner = ResumableDocumentScanner()
+            var nextIndex = 0
+
+            while let chunk = try reader.readChunk() {
+                let completed = try scanner.consume(chunk)
+                for source in completed {
+                    try emit(source, index: &nextIndex, body)
+                }
+            }
+
+            if let source = try scanner.finish(), !source.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty {
+                try emit(source, index: &nextIndex, body)
+            }
+        }
+
+        private func emit(
+            _ source: String,
+            index: inout Int,
+            _ body: (PureYAML.Stream.Document) throws -> Void,
+        ) throws {
+            let documents = try pureParser.parseStream(source)
+            for document in documents {
+                try body(.init(index: index, value: document.value))
+                index += 1
+            }
+        }
+    }
+}

Sources/PureYAMLStreaming/PureYAMLStreaming.swift

@@ -0,0 +1,2 @@
+/// Root namespace for PureYAML streaming extensions.
+public enum PureYAMLStreaming {}

Sources/PureYAMLStreaming/ResumableDocumentScanner.swift

@@ -0,0 +1,86 @@
+import Foundation
+
+extension PureYAMLStreaming {
+    /// Resumable document-boundary scanner.
+    ///
+    /// This is intentionally a document-token scanner, not yet a YAML grammar
+    /// token scanner. It proves chunked input and document-at-a-time parsing
+    /// while the lower-level token scanner remains incubating.
+    public struct ResumableDocumentScanner: Sendable {
+        private var pendingLine = ""
+        private var currentDocument = ""
+        private var hasCurrentDocumentContent = false
+        private var line = 1
+
+        public init() {}
+
+        public mutating func consume(_ chunk: String) throws -> [String] {
+            var documents: [String] = []
+            pendingLine.append(chunk)
+
+            while let newlineRange = pendingLine.rangeOfCharacter(from: .newlines) {
+                let lineText = String(pendingLine[..<newlineRange.lowerBound])
+                let newline = String(pendingLine[newlineRange])
+                pendingLine.removeSubrange(..<newlineRange.upperBound)
+                try consumeLine(lineText + newline, documents: &documents)
+                line += 1
+            }
+
+            return documents
+        }
+
+        public mutating func consumeToken(_ chunk: String) throws -> [StreamingToken] {
+            try consume(chunk).map(StreamingToken.documentSource)
+        }
+
+        public mutating func finish() throws -> String? {
+            if !pendingLine.isEmpty {
+                var ignored: [String] = []
+                try consumeLine(pendingLine, documents: &ignored)
+                pendingLine.removeAll(keepingCapacity: true)
+            }
+            defer {
+                currentDocument.removeAll(keepingCapacity: true)
+                hasCurrentDocumentContent = false
+            }
+            return currentDocument.isEmpty ? nil : currentDocument
+        }
+
+        private mutating func consumeLine(
+            _ sourceLine: String,
+            documents: inout [String],
+        ) throws {
+            if isDocumentStart(sourceLine), hasCurrentDocumentContent {
+                documents.append(currentDocument)
+                currentDocument = sourceLine
+                hasCurrentDocumentContent = true
+                return
+            }
+
+            currentDocument.append(sourceLine)
+            if hasSemanticContent(sourceLine) {
+                hasCurrentDocumentContent = true
+            }
+        }
+
+        private func isDocumentStart(_ sourceLine: String) -> Bool {
+            let trimmedNewline = sourceLine.trimmingCharacters(in: .newlines)
+            guard trimmedNewline.hasPrefix("---") else {
+                return false
+            }
+            let markerEnd = trimmedNewline.index(trimmedNewline.startIndex, offsetBy: 3)
+            if markerEnd != trimmedNewline.endIndex {
+                let next = trimmedNewline[markerEnd]
+                guard next == " " || next == "\t" || next == "#" else {
+                    return false
+                }
+            }
+            return trimmedNewline.prefix(3) == "---"
+        }
+
+        private func hasSemanticContent(_ sourceLine: String) -> Bool {
+            let trimmed = sourceLine.trimmingCharacters(in: .whitespacesAndNewlines)
+            return !trimmed.isEmpty && !trimmed.hasPrefix("#")
+        }
+    }
+}

Sources/PureYAMLStreaming/StreamingError.swift

@@ -0,0 +1,21 @@
+import PureYAML
+
+extension PureYAMLStreaming {
+    /// Errors from the chunked streaming layer before or around PureYAML parsing.
+    public enum StreamingError: Error, Equatable, Sendable, CustomStringConvertible {
+        case invalidUTF8(line: Int, column: Int, index: Int)
+        case invalidChunkSize(Int)
+        case nestedDocumentStart(line: Int, column: Int)
+
+        public var description: String {
+            switch self {
+            case let .invalidUTF8(line, column, index):
+                "invalid UTF-8 at line \(line), column \(column), byte \(index)"
+            case let .invalidChunkSize(size):
+                "chunk size must be greater than zero, got \(size)"
+            case let .nestedDocumentStart(line, column):
+                "document start marker found inside an open collection at line \(line), column \(column)"
+            }
+        }
+    }
+}

Sources/PureYAMLStreaming/StreamingMark.swift

@@ -0,0 +1,16 @@
+extension PureYAMLStreaming {
+    /// Source position at a streaming boundary.
+    public struct StreamingMark: Equatable, Sendable {
+        public var line: Int
+        public var column: Int
+        public var index: Int
+
+        public init(line: Int, column: Int, index: Int) {
+            self.line = line
+            self.column = column
+            self.index = index
+        }
+
+        public static let start = StreamingMark(line: 1, column: 1, index: 0)
+    }
+}

Sources/PureYAMLStreaming/StreamingToken.swift

@@ -0,0 +1,6 @@
+extension PureYAMLStreaming {
+    /// Tokens emitted by the current streaming scanner milestone.
+    public enum StreamingToken: Equatable, Sendable {
+        case documentSource(String)
+    }
+}

Tests/PureYAMLStreamingTests/StreamingTests.swift

@@ -0,0 +1,99 @@
+import Foundation
+import PureYAML
+@testable import PureYAMLStreaming
+import Testing
+
+@Suite("PureYAML Streaming")
+struct StreamingTests {
+    @Test("chunked reader preserves UTF-8 split across chunks")
+    func chunkedReaderPreservesSplitUTF8() throws {
+        var reader = try PureYAMLStreaming.ChunkedUTF8Reader(
+            data: Data("aé\nb".utf8),
+            chunkSize: 2,
+        )
+
+        var chunks: [String] = []
+        while let chunk = try reader.readChunk() {
+            chunks.append(chunk)
+        }
+
+        #expect(chunks.joined() == "aé\nb")
+        #expect(reader.mark == .init(line: 2, column: 2, index: 5))
+    }
+
+    @Test("document scanner emits documents across chunk boundaries")
+    func scannerEmitsDocumentsAcrossChunks() throws {
+        var scanner = PureYAMLStreaming.ResumableDocumentScanner()
+        var documents: [String] = []
+
+        documents += try scanner.consume("---\na: 1\n--")
+        documents += try scanner.consume("-\nb: 2\n")
+        if let final = try scanner.finish() {
+            documents.append(final)
+        }
+
+        #expect(documents == ["---\na: 1\n", "---\nb: 2\n"])
+    }
+
+    @Test("parser streams documents in order")
+    func parserStreamsDocumentsInOrder() throws {
+        let yaml = Data(
+            """
+            ---
+            name: one
+            ---
+            name: two
+            """.utf8,
+        )
+        let parser = PureYAMLStreaming.Parser(chunkSize: 5)
+        let documents = try parser.collectDocuments(data: yaml)
+
+        #expect(documents.count == 2)
+        #expect(documents[0].index == 0)
+        #expect(documents[1].index == 1)
+        #expect(stringValue("name", in: documents[0].value) == "one")
+        #expect(stringValue("name", in: documents[1].value) == "two")
+    }
+
+    @Test("streaming matches PureYAML on Geekbench fixtures")
+    func streamingMatchesGeekbenchFixtures() throws {
+        let fixtureRoot = URL(fileURLWithPath: #filePath)
+            .deletingLastPathComponent()
+            .deletingLastPathComponent()
+            .deletingLastPathComponent()
+            .deletingLastPathComponent()
+            .appendingPathComponent("PureYAMLGeekbench")
+            .appendingPathComponent("Fixtures")
+            .appendingPathComponent("real-yaml")
+        guard FileManager.default.fileExists(atPath: fixtureRoot.path) else {
+            return
+        }
+
+        let files = try FileManager.default.contentsOfDirectory(
+            at: fixtureRoot,
+            includingPropertiesForKeys: nil,
+        ).filter { url in
+            url.pathExtension == "yaml" || url.pathExtension == "yml"
+        }
+
+        #expect(files.count >= 100)
+
+        let streamingParser = PureYAMLStreaming.Parser(chunkSize: 4096)
+        let pureParser = PureYAML.Parsing.Parser()
+        for file in files {
+            let data = try Data(contentsOf: file)
+            let streamed = try streamingParser.collectDocuments(data: data).map(\.value)
+            let direct = try pureParser.parseStream(String(decoding: data, as: UTF8.self)).map(\.value)
+            #expect(streamed == direct)
+        }
+    }
+
+    private func stringValue(_ key: String, in value: PureYAML.Model.Value) -> String? {
+        guard case let .mapping(mapping) = value,
+              case let .string(string)? = mapping[key]
+        else {
+            return nil
+        }
+        return string
+    }
+}

docs/ARCHITECTURE.md

@@ -0,0 +1,46 @@
+# Streaming Architecture
+
+PureYAMLStreaming separates three concerns:
+
+1. `ChunkedUTF8Reader` pulls bounded byte chunks from a file or data source and
+   tracks byte offsets plus line and column marks.
+2. `ResumableDocumentScanner` consumes chunks, detects top-level YAML document
+   boundaries, and emits one document source at a time.
+3. `Parser` parses each emitted document through PureYAML and hands completed
+   documents to the caller.
+
+```mermaid
+flowchart TD
+    fileSource["File or Data"] --> chunkedReader["Chunked UTF8 Reader"]
+    chunkedReader --> documentScanner["Resumable Document Scanner"]
+    documentScanner --> pureParser["PureYAML Parser"]
+    pureParser --> documentSink["Document Sink"]
+```
+
+## Why Document Streaming First
+
+PureYAML's internal parser already avoids retaining the full token and event
+arrays, but the public boundary is still a complete `String`. A fully resumable
+YAML token scanner needs careful handling for quoted scalars, block scalars,
+comments, indentation, CRLF, Unicode, tags, anchors, and merge keys.
+
+Document streaming is the safe first milestone: it proves file/chunk IO,
+document-at-a-time memory behavior, and API ergonomics without forking the YAML
+grammar implementation.
+
+## Future Event Streaming
+
+The next scanner milestone should replace `ResumableDocumentScanner` with a
+token-level scanner that can suspend and resume inside any YAML token. That
+scanner can then feed a public event sink directly.
+
+The intended final shape is:
+
+```mermaid
+flowchart TD
+    bytes["Async Bytes"] --> utf8Reader["Chunked UTF8 Reader"]
+    utf8Reader --> tokenScanner["Resumable Token Scanner"]
+    tokenScanner --> eventParser["Token Event Parser"]
+    eventParser --> eventSink["Event Sink"]
+    eventParser --> documentComposer["Optional Document Composer"]
+```

scripts/check.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+swift build --package-path "$ROOT_DIR"
+swift test --package-path "$ROOT_DIR"