116 lines
5.3 KiB
Markdown
116 lines
5.3 KiB
Markdown
# PureYAMLStreaming
|
|
|
|
[](https://git.aleahim.com/NodetteHQ/PureYAMLStreaming/actions?workflow=macos.yml)
|
|
[](https://git.aleahim.com/NodetteHQ/PureYAMLStreaming/actions?workflow=linux.yml)
|
|
[](https://git.aleahim.com/NodetteHQ/PureYAMLStreaming/actions?workflow=windows.yml)
|
|
[](https://git.aleahim.com/NodetteHQ/PureYAMLStreaming/actions?workflow=wasm.yml)
|
|
[](https://git.aleahim.com/NodetteHQ/PureYAMLStreaming/actions?workflow=wasm-site.yml)
|
|
|
|
Experimental full-streaming incubator for PureYAML.
|
|
|
|
This package is intentionally separate from `PureYAML` while the chunked input,
|
|
document streaming, and eventual event streaming APIs settle. It depends on
|
|
PureYAML for the stable model and parser semantics, then adds streaming input
|
|
adapters around that stable core.
|
|
|
|
## Status
|
|
|
|
Prototype.
|
|
|
|
The current implementation streams YAML input in chunks, detects document
|
|
boundaries with PureYAML's resumable token scanner
|
|
(`PureYAML.Streaming.Scanner`), and emits one parsed
|
|
`PureYAML.Stream.Document` at a time. That gives bounded memory for
|
|
multi-document streams where each document fits in memory.
|
|
|
|
Boundary detection is grammar-driven, not line-based: a `---` counts as a
|
|
boundary only where the token scanner emits a `documentStart` token, a
|
|
directive starts a new document only after an explicit `...`, and a chunk
|
|
boundary may fall anywhere, including mid-token. Splitting is lossless; the
|
|
emitted document sources concatenate back to the exact input.
|
|
|
|
A future milestone will surface token-level events to event-level consumers
|
|
directly; the public `StreamingToken` currently carries whole document
|
|
sources only.
|
|
|
|
## Memory Contract
|
|
|
|
- `parseDocuments` is bounded by the largest current document plus parser state.
|
|
- `collectDocuments` is a convenience helper and retains all returned documents.
|
|
- Anchors, aliases, merge keys, and `PureYAML.Model.Value` construction can
|
|
require retaining semantic state until the current document is complete.
|
|
- Future event-level APIs should be lower memory than document-level APIs, but
|
|
they still cannot promise constant memory for every YAML feature.
|
|
|
|
## Example
|
|
|
|
```swift
|
|
import Foundation
|
|
import PureYAMLStreaming
|
|
|
|
let url = URL(fileURLWithPath: "stream.yaml")
|
|
let parser = PureYAMLStreaming.Parser()
|
|
|
|
try parser.parseDocuments(from: url) { document in
|
|
print("document", document.index, document.value)
|
|
}
|
|
```
|
|
|
|
## WASM Browser Benchmark Site
|
|
|
|
Live site:
|
|
|
|
`https://mihaela.codeberg.page/PureYAMLStreaming/`
|
|
|
|
Deployed from the `pages` branch of the public repository
|
|
`https://codeberg.org/Mihaela/PureYAMLStreaming`, which holds only the built site
|
|
(this repository's own `pages` branch is a copy): `bash scripts/build-wasm-site.sh`,
|
|
copy `WasmSite/` (with the gzipped module, not the raw one) onto that branch, and push
|
|
it to the Codeberg repository; a webhook there triggers the git-pages redeploy.
|
|
|
|
Build the Swift WASI benchmark executable and copy it into the static test site:
|
|
|
|
```sh
|
|
bash scripts/build-wasm-site.sh
|
|
```
|
|
|
|
The build script requires `npm` so it can stage the pinned browser WASI shim as
|
|
a local `WasmSite/vendor/` asset. The deployed page does not import the shim from
|
|
a runtime CDN.
|
|
|
|
Serve the site locally:
|
|
|
|
```sh
|
|
bash scripts/serve-wasm-site.sh
|
|
```
|
|
|
|
Then open `http://localhost:8080` and click **Run Buffered Benchmark** or
|
|
**Run True Streaming Benchmark**. The page loads
|
|
`pureyaml-streaming-wasm-smoke.wasm`, lets you choose from 10+ public
|
|
multi-megabyte YAML files, and reports fetch time, Swift parse time,
|
|
throughput, document count, stdout JSON, and a timestamped run log.
|
|
|
|
Inside Swift, `SwiftWASIHTTPClient.HostHTTPClient` models that host-provided
|
|
HTTP response. The benchmark then parses the response body through
|
|
`PureYAMLStreaming.ChunkedUTF8Reader` and `ResumableDocumentScanner`, reporting
|
|
chunk count, chunk size, scanner document-source emissions, fetch time, parse
|
|
time, throughput, document count, and stdout JSON.
|
|
|
|
The buffered benchmark fetches the selected YAML file in JavaScript before
|
|
invoking the WASM guest. It keeps `SwiftWASIHTTPClient.HostHTTPClient` in the
|
|
loop to model the host-response boundary.
|
|
|
|
The experimental true-streaming benchmark requires `crossOriginIsolated`,
|
|
`SharedArrayBuffer`, `Worker`, and `Atomics.wait`. A same-origin COOP/COEP
|
|
service worker enables that mode on localhost and Codeberg Pages. In this path,
|
|
the main thread reads `fetch().body` with a `ReadableStream` reader, writes each
|
|
network chunk into a `SharedArrayBuffer` ring buffer, and a Worker-owned WASI
|
|
runtime serves those bytes through a blocking custom stdin `fd_read`. Swift then
|
|
parses stdin with `ChunkedUTF8Reader` without calling `readDataToEndOfFile()`.
|
|
|
|
## Relationship To PureYAML
|
|
|
|
`PureYAML` remains the dependency-free core package. This package is the
|
|
incubator for file, byte, and async streaming APIs. Stable pieces can move back
|
|
into PureYAML later once the API and memory behavior are proven.
|