Skip to content

feat: Muse gated passthrough stage 1 (measurement + allow/deny) - #17

Draft
chaodu-agent wants to merge 6 commits into
mainfrom
feat/muse-stage1
Draft

chaodu-agent wants to merge 6 commits into
mainfrom
feat/muse-stage1

Conversation

@chaodu-agent

Copy link
Copy Markdown
Contributor

Summary

Stage 1 of the Muse gated passthrough: measurement only, plus allow/deny passthrough.

  • Muse custom connectors send the full Discord HTTP request to a public listener here, with the caller key in a header. The key is not exchanged.
  • The proxy decides allow or deny, forwards allowed requests upstream, and returns the upstream response unchanged.
  • Each decision is written to an evidence log. A probe path is available for connectivity checks.
  • No approval hold: that is stage 2.

Tested

  • cargo fmt --check, cargo clippy, cargo test
  • Black-box passthrough matrix
  • The new ci.yml steps
  • Stage 1 acceptance tests
  • Live pilot verification on a homelab host

Deferred to stage 2

  • Holding the connection until a human approves
  • Any approval proof bound to the request hash
  • Muse-side latency and hold-time measurements

Note: the in-flight port PRs also touch config.rs, audit.rs and pipeline.rs. This branch is based on current origin/main, so expect a rebase on whichever lands second.

…passthrough pipeline, public listener, evidence, probe)

Verified on macmini (rsync of worktree): cargo clippy --workspace --all-targets -D warnings clean;
cargo clippy -p ap-proxy --features stage1-probe,matrix-upstream clean; cargo test --workspace all pass;
cargo test -p ap-proxy --lib --features stage1-probe: 61 pass. Black-box matrix, docs and CI follow.
… for muse stage 1

Ran on macmini (rsync of worktree, PATH with homebrew openssl 3):
- cargo fmt --all --check: ok
- cargo clippy --workspace --all-targets -D warnings: clean; same with -p ap-proxy --features stage1-probe,matrix-upstream
- cargo test --workspace: all pass (ap-core 176, ap-proxy lib 57, 61 with stage1-probe, others unchanged)
- release build (default features): no AP_TEST_*, __ap_probe or __ap_test_panic strings; ap-egress does not depend on ap-broker
- python3 tests/matrix/passthrough.py: summary 13 PASS of 13 rows; cargo test --test matrix_passthrough ok
- sidecar python3 tests/matrix/matrix.py: 41 PASS of 41 rows under /usr/bin/python3. Under Homebrew python 3.14 + OpenSSL 4, row R4-2 fails in the harness client (certificate verify: missing AKI), unrelated to this change. The existing cargo test --test matrix cannot find its binaries with cargo 1.100's split build dir (pre-existing, left unchanged).
- deploy/k8s/test_validate_manifests.py not run: no pyyaml on macmini (files untouched).
- passthrough audit record gets bytes_sent: None (field added by #16)
- drop dead __ap_probe reserved-name check; mark caller key header values sensitive
- evidence digest comment says what the digest covers
- PT-12: obs-fold row, GET CL+TE and bare-LF measured as known gap
- design 3.1/3.5/acc 12 and runbook: hyper behaviour, loopback + front reliance, stage 2 raw-head scanner

Checked on macmini: fmt, clippy (default and stage1-probe,matrix-upstream) -D warnings, cargo test --workspace, release hook greps clean, sidecar matrix 41/41, passthrough matrix 13/13, matrix_passthrough cargo target ok. cargo matrix target fails on this host only (new cargo test-exe layout; bin_dir lookup), driver run directly instead.
@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Round 2, head 47605fe (still a draft).

  • Merged origin/main (docs: kiro-cli X-Amz-Target evidence and kiro-cps route #14, fix: remove egress.allow_route_hosts #15, feat: bytes_sent on upstream_reset audit line, with H2 upstream tests #16) with a merge commit; no textual conflicts. One semantic fix: the passthrough audit record needed the new bytes_sent field (None).
  • Framing: hyper 1.12 deletes Content-Length when Transfer-Encoding: chunked is present, collapses identical duplicate Content-Length, and accepts bare LF, so the handler cannot refuse those three. The gap is documented (design 3.1, 3.5, acceptance 12, runbook) and deferred. Stage 1 relies on the loopback-only listener plus the measured normalisation by the front. A connection-level raw-head scanner is the stage 2 candidate, to be decided before any listener is exposed beyond the front. Option A (the scanner) was not built: a second parser in front of hyper adds a parser-differential surface, and disabling keep-alive would change behaviour the front's pooled connections rely on. The owner should be told it was not built.
  • Review fixes: obs-fold row added to PT-12 (hyper answers 400); dead __ap_probe reserved-name check removed; caller-key header values marked sensitive; evidence digest comment corrected. Skipped: GET-only grant guard, because the existing passthrough tests and matrix rows grant POST on purpose.
  • Verified on macmini: fmt, clippy -D warnings (default and stage1-probe + matrix-upstream), workspace tests, release hook greps clean, sidecar matrix 41/41, passthrough matrix 13/13, matrix_passthrough cargo target. The matrix cargo target fails on that host only (newer cargo puts test executables in the build dir, so its binary lookup misses); the driver was run directly instead.
  • The binary deployed on p1 is still the old one.

…nd 3 review fixes

Verified on macmini: fmt, clippy (default, stage1-probe, stage1-probe+matrix-upstream), workspace tests,
stage1-probe lib tests (66), sidecar matrix 41/41, passthrough matrix 15/15, release hook greps all 0.
@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Follow-up to my review of 47605fe, fixed in 4ed4388.

Finding Result
MEDIUM: response Connection nominations were forwarded Fixed. response_headers now collects every token from all upstream Connection values (case-insensitive, trimmed) and drops those fields, plus a response-only set containing Proxy-Authenticate. Unit test response_drops_hop_headers_and_connection_nominations; matrix rows PT-15 (Connection: X-Hop + X-Hop) and PT-16 (Proxy-Authenticate). Passthrough matrix is now 15 rows.
LOW 1: caller_key_header_present and host_matches_public_authority were always true Fixed. Both fields removed from evidence; runbook now says a wrong Host shows only as a host_rejected audit line.
LOW 2: writes denied only by policy content Skipped. It matches design acceptance 13 and the pilot policy is GET-only; refusing non-GET allow grants would change behaviour, so I left it for the owner to decide.
LOW 3: probe body read had no total deadline Fixed. The read loop is now bounded by request_deadline_secs in addition to the per-chunk limit.
LOW 4: CI did not lint/test stage1-probe Fixed. ci.yml adds clippy and cargo test -p ap-proxy --features stage1-probe --lib.
LOW 5: [M] source for the framing gap not in the repo Fixed. Redacted excerpt added at docs/evidence/muse-stage1-front-measurements.md (no hostnames, addresses, logins, URLs or keys); the design doc cites it.
LOW 6: runbook nits Fixed. Revocation now says to delete the identity and its grants (and why); probe sentence reworded.

The framing gap (identical duplicate Content-Length, bare LF, Content-Length + chunked) is unchanged and stays documented, not built.

Run on macmini: fmt, clippy (default, stage1-probe, stage1-probe+matrix-upstream) with -D warnings, cargo test --workspace, stage1-probe lib tests (66), sidecar matrix 41/41, passthrough matrix 15/15, release-binary hook greps (all 0), ap-egress dependency check. Nothing was redeployed.

…every refusal body

- passthrough: caller key from the header OR the /ap/<key>/ prefix (parse_ap_path),
  exactly one source (both: 400 caller_key_ambiguous); prefix stripped before
  routing/forwarding; probe works under the path form
- no-leak tests and matrix rows extended to the path form (PT-06), new PT-17,
  table-driven test over every refusal row (status, header, exact body)
- docs: design 3.1/3.2/3.5/stage 1/open questions, runbook

Verified on macmini: fmt --check, clippy (default, stage1-probe, matrix-upstream+stage1-probe)
-D warnings, cargo test --workspace, ap-proxy stage1-probe --lib (74), release hook greps clean,
sidecar matrix.py 41/41, passthrough matrix 16/16 (cargo target matrix_passthrough ok).
The sidecar cargo target 'matrix' fails on this host only on its binary-dir lookup (cargo build-dir layout), file untouched.
@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Round 3 pushed (d108ceb), after Muse-reported facts about the Discord helper (not measured by us).

Code (additive; no existing check weakened or reordered)

  • Passthrough routes accept the caller key from X-AP-Caller-Key or a /ap/<caller-key>/<route>/<rest> prefix (existing parse_ap_path, same key format). Exactly one source: both present is 400 caller_key_ambiguous. The key is still compared by sha256 in constant time, and the prefix is stripped before routing and forwarding. The probe works under the path form. Authorization is not a source, since it is the upstream credential and is forwarded unchanged.
  • The key and the /ap/ prefix do not appear in any audit, evidence or journal line, error body or panic output. Unit tests and the passthrough matrix (PT-06) now cover the path form, including 4xx/5xx and panic paths.
  • Every refusal body is {"ap_decision":"<decision>","code":"<code>"}, in addition to the header, because the helper drops response headers. A table-driven test checks status, header and exact body for every refusal row in both key forms. The matrix refused() helper also checks the body decision.

Docs: design 3.1, 3.2, 3.5, stage 1 and open questions, plus a short [M-reported] facts list. The 30 s JSON timeout is recorded as a hard limit on the stage 2 hold window. The gate constrains the agent only if the new connector is its only route. Runbook updated for both key sources.

Verification (macmini): fmt, clippy (default, stage1-probe, matrix-upstream+stage1-probe) with -D warnings, cargo test --workspace, release-binary hook greps clean, sidecar matrix 41/41, passthrough matrix 16/16 (was 15; PT-17 added). The sidecar cargo target matrix fails on that host only while looking up its binary directory (a cargo build-dir layout quirk). That file is untouched, and the driver run directly passes 41/41.

Known gap unchanged: strict rejection of duplicate identical Content-Length, bare LF and CL+TE is documented, not built. Nothing was deployed or restarted. The PR stays a draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant