RFC-022 Stream descriptors and view-time pipes

RFC 022: Stream descriptors and view-time pipes

Status: Accepted
Date: 2026-09-23
Topics: Workspace config, pack manifest, grid layout, schema neutrality
Repos: vyasa (packer + manifest), vyasa-apps (grid pipes), vyutils (engines)

Does not revise: RFC 006 (stale .vypkg/.db package sketch), RFC 013 (projection profiles), RFC 015 (structure/views/URNs), RFC 017 (ecosystem glossary), RFC 018 (embedded viewer). Those documents do not define per-stream descriptors or view-time pipes. Workspace history in notes/vyasa-workspace.md is a changelog, not an RFC.

Cites: RFC 017 for recommended kind tokens (source / commentary / translation). Invariant 1.2 (schema-neutrality). Invariant 1.6 (packed stream ids vs primary = true).

---

1. Problem

Readers want derived grid columns such as mula | lipi(Mlym) and padapatha | krama without the publisher packing Malayalam or Krama streams. Language/wording stays a pack decision. Script and recitation mode can be view-time pipes of packed bytes.

Today the viewer would have to guess script and text type. [catalog] language is one BCP 47 for the work. layout = sequence | document is addressing, not genre. Putting detect_script, vyasa-lipi::Script, or kind = pada ⇒ krama into vyasac would leak Indic/Vedic behavior into the compiler.

A packed transform allow-list would force every work to be repacked when the app ships a new engine. That is app policy, not a fact about the bytes.

2. Decision

1. Every content stream declares opaque language, script, and kind.
2. vyasac stores and syntax-checks those strings. It does not interpret engines, scripts, or kinds, and it does not pack a transforms list.
3. Whoever writes the pipe (grid customizer, later a packed view template) names the engines. New engines work on already-packed .vyview files.
4. Viewer + engines map packed script to an engine from when the token is in that engine’s table. Unknown engine or unsupported from → customizer / diagnostics error (do not silently show source). WASM missing → show the source cell; never guess script.
5. Grid cells may name a pipe whose left-hand side is a packed stream id. Weave still keys HTML by that id. Pipes run in JS after weave, using @project-vyasa/sanskrit-wasm (lazy), never vyasav_bg.wasm.

3. stream.toml (not workspace [streams])

vyasac.toml has no [streams] table. Leftover [streams] is a load error. Pack discovers immediate content/<folder> directories; packed id = folder name. Each folder must contain stream.toml. Missing any of language, script, kind is a pack error (same posture as content_themes). Loose content/*.vy is a pack error.

Do not name this sidecar vyasac.toml (collides with workspace config). context.vy stays language inheritance, not pack facts: a missing or stringly stream.name is harder to diagnose than a required Toml file next to the .vy files.

content/samhita/stream.toml

language = "sa"                 # BCP 47; syntax-check only
script = "Deva"                 # opaque; ISO 15924 preferred when it exists
kind = "source"                 # opaque; recommended tokens below
primary = true                  # URN spine; exactly one packed stream

# content/padapatha/stream.toml
language = "sa"
script = "Deva"
kind = "source"

# content/sayana/stream.toml
language = "sa"
script = "Deva"
kind = "commentary"

# content/translation/stream.toml
language = "en"
script = "Latn"
kind = "translation"

Packed IAST of the same wording is still language = "sa", kind = "source", script = "IAST" (not English). A Bible translation is language = "en", script = "Latn", kind = "translation".

primary = true is only the URN-spine flag (Invariant 1.6). Packed / runtime id is the folder. Do not add kind = "primary" — that collides with the pack-time alias (pack error). Nested content/transliteration/iast packs as stream transliteration (depth-1 folder). Annotations and vocabulary live outside content/.

3.1 What vyasac may check

Field	Rule in Rust
language	Non-empty; BCP 47 shape (^[a-z]{2,3}(-[A-Za-z0-9]+)*$), not a language registry
script	Non-empty; ^[A-Za-z][A-Za-z0-9-]{1,15}$
kind	Non-empty; ^[a-z][a-z0-9-]{0,31}$; value primary is rejected (spine-alias collision)
primary	Optional bool, default false. Exactly one packed stream must be true
segment_separator	Optional string

vyasac must not:

• Depend on vyasa-lipi / vyasa-patha
• Call detect_script (including --strict)
• Map kind → legal pipes
• Treat pada, deva, malayalam, or krama as compiler vocabulary
• Infer layout = sequence | document from kind
• Pack or allow-list engine tokens

Unknown kind tokens (other than primary) are legal at pack. The layout author and the engine decide whether a pipe can run.

3.2 Recommended tokens (docs and engines, not Rust)

kind (RFC 017 layers, lowercase):

Token	Meaning
source	Spine wording of this stream (mula, saṃhitā, Hebrew text, …)
commentary	Explanation aligned to the spine
translation	Wording in another language

Do not pack kind = "pada". Pada-pāṭha is a source stream whose files happen to be pada text; recitation is a view-time pipe the layout names.

Do not pack kind = "primary". Spine authority is primary = true on exactly one stream.toml.

script: prefer ISO 15924 (Deva, Mlym, Latn, Hebr, Grek) when it names the writing system. Sanskrit romanizations that ISO 15924 does not distinguish from Latin use extra tokens IAST and ISO15919. Mapping those strings to an engine is the engine’s table, not the packer.

Pipes are not Toml. The first engine registry lives in the viewer and Sanskrit WASM (lipi, krama, jata, …). Latn is not a lipi source lives in the lipi engine (no from named Latn / latn). A new engine ships in the app; old packs do not need a rebuild.

Translation is not a pipe. The reader shows a kind = translation stream. Do not LLM.

Chrome localization (structure.chapter = अध्याय) stays in localization.vy. It is labels, not kind.

4. Packed contract

Write a compact JSON object on the view manifest, key streams_meta, keyed by packed stream name:

{
  "samhita": { "language": "sa", "script": "Deva", "kind": "source" },
  "padapatha": { "language": "sa", "script": "Deva", "kind": "source" },
  "sayana": { "language": "sa", "script": "Deva", "kind": "commentary" }
}

Do not add descriptor columns to the streams table (id, name only). vyasav exposes this JSON to JS; it does not execute pipes. vyasav weave --layout diagnostics stay about keyed/scrape/missing source streams.

5. Grid pipes

Default layout remains one column per packed stream. Pipes are opt-in in the customizer (later: packed view templates). The layout expression is the declaration of which engines run.

Layout JSON (weave still sees only block):

{ "block": "mula", "pipe": ["lipi:Mlym"] }

pipe omitted or [] = today’s cell. Each step is engine or engine:Arg. Customizer text mula | lipi(Mlym) and padapatha | krama | lipi(Telu) parses into block + pipe. weave_layout does not parse pipes and does not grow WASM.

Legality (viewer / engine, not pack):

lipi(to)   iff the lipi engine maps stream.script → from
           and the lipi engine maps `to` → a script
krama|jata iff that engine is in the app registry
unknown engine token → customizer / diagnostics error (do not silently show source)
WASM missing → show source cell (or placeholder); never guess script

mula | krama is a layout choice. The engine may produce a poor column if the bytes are not pada-pāṭha; the packer does not veto it. Publishers who know their streams put padapatha | krama in a view; readers who type an inapplicable pipe see an engine error or a bad column.

CSS: keep .vyasa-block-{packedId} on the source identity. Add .vyasa-pipe-{engine} (and data-pipe). Do not invent .vyasa-block-mula-malayalam (collides with a real folder).

Column name (a11y): “mula (Mlym)” / “padapatha (krama)” — announce a view of the packed stream.

Pipeline order is user-written order. Prefer generate_krama_in_script when the last step is lipi so sandhi/iti stay in one engine.

HTML policy (v1, locked):

Pipe	How
lipi	Walk text nodes in the keyed HTML; preserve tags and classes
krama / jata	Take innerText of the keyed cell, generate, wrap in <div class="vyasa-block-{id} vyasa-pipe-krama">

If Krama quality is bad because the HTML is mixed, follow BACKLOG §13 (pack-time keyed plaintext). Do not scrape inner classes in this RFC. Do not add detect_script on the read path.

Reading view (weave_view) is out of slice until grid pipes are honest.

6. layout = sequence | document is orthogonal

That flag is addressing / extraction, not text type. Do not infer it from kind. Publication-level layout stays for v1. Per-stream addressing is a later RFC revision if needed.

	sequence	document
kind = source	BG mula, RV samhita	Prose work, no verse markers
kind = commentary	Commentary aligned per verse	Continuous file

7. Chrome vs grid

A global “Script: Malayalam” for sidebar titles uses the spine packed stream’s script as from (lookup via Invariant 1.6 primary_stream). It is not the same as adding a grid column mula | lipi(Mlym). Share one preference store later; do not block slice 0–4.

8. Layer table

Layer	Does
vyasac	Require language / script / kind; syntax-check; write streams_meta
vyasav	Pass JSON through; keyed HTML by packed id; no lipi/patha; observational load/weave contract only (vyasa/assessments/vyasav/LOAD_INTEGRITY.md)
vyasa-apps grid	Parse pipe; consult packed script / engine tables; post-weave rewrite; lazy WASM
Sanskrit WASM	transliterate(from=packed.script, to=…); generate_krama / generate_krama_in_script

9. Non-goals

• Packing a Malayalam or Krama stream for every work
• A packed transforms allow-list (would force republish for new engines)
• Pipes inside weave_view / {{ body }} in v1
• Deprecating localization.vy / extend
• Putting patha/lipi in vyasav
• Closed script/kind enums in Rust
• Runtime translation / LLM
• Inferring sequence vs document from kind
• translation_of pairing (later)
• Pack-time detect_script

10. Implementation slices

0. Packer: content/<folder>/stream.toml, exactly one primary = true, pack errors, streams_meta, tests (missing toml; leftover [streams]; opaque unknown kind still packs; kind = "primary" fails; packed-name keys). Drop workspace [streams] / path. Document in workspace-config.
1. Viewer: read streams_meta; Language / kind filters may use it before pipes exist.
2. Grid parser: block + pipe; unknown engine / lipi from miss → UI error.
3. lipi column with packed from.
4. krama / jata via innerText; then krama | lipi.
5. Chrome Script using spine script; optional packed view templates with pipes.

11. Revision history

• v1 (2026-09-23): Accepted. Opaque pack fields; engine policy in viewer; kind not role; no pada in the compiler; grid-first pipes; packed transforms[].
• v2 (2026-09-23): Drop packed transforms. Pipes are declared in the layout. language / script / kind remain pack-time facts so the viewer never detect_script.
• v3 (2026-09-24): Stream facts in content/<folder>/stream.toml. Drop [streams] in vyasac.toml. URN spine is primary = true on exactly one stream. Packed id is always the folder name.