v1.0.663 — Sentinel Policy M1: immutable ingestion spine

Context

Sentinel Policy is a new Sentinel Signal Systems product (policy.sentinelsignal.io), a healthcare policy-change intelligence platform. The user supplied a 207-requirement architecture spec (saved as the sentinel-policy-architecture-plan project memory) and asked to go ahead with the implementation. Building all 12 milestones in one pass isn't realistic — the spec itself mandates milestone-gated delivery (AGENT-001: "do not skip ahead"), and M0's own exit criterion blocks any commercial/customer-facing data endpoint until a source-terms/licensing review happens, which this pass does not do. Scoped explicitly to M1 only: source registry, scheduler, secure bounded fetcher, immutable object archive, source-health model. Confirmed with the user before starting.

What shipped

A new policy/ subproject inside this same repo (not a separate monorepo), mirroring verify/'s structure and conventions 1:1: pyproject.toml, src/sentinel_policy/, Alembic migrations, Dockerfile, tests/. Package sentinel_policy.

Data model (db/models.py, alembic/versions/0001_initial.py): Source (SRC-001/002/004/006

— identity, org, program, jurisdiction, allowlisted hostnames, adapter config, cadence, terms-review state, health status), SourceDiscoveryRun (ING-001), DiscoveredResource, DocumentAsset (RAW-001..006 — content-addressed by SHA-256, never overwritten), DocumentObservation (ING-002 — full HTTP metadata per fetch).

Object storage (storage.py): ObjectStore Protocol with S3ObjectStore (boto3, IONOS

S3-compatible, reuses the same AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY credentials already provisioned for Postgres backups, pointed at a new bucket) and LocalFilesystemObjectStore (no external dependency — selected automatically when SENTINEL_POLICY_S3_BUCKET is unset, used for local dev and all tests). Put-by-hash, immutable — refuses to overwrite an existing key.

Secure fetcher (fetch.py, SECURITY-001/002/003, ING-002..007): per-source hostname

allowlist, SSRF defense (resolves DNS itself, rejects private/loopback/link-local addresses, re-validated on every redirect hop — not just the initial URL), size/timeout caps, MIME validation via header + content-sniffing, bounded retries with jitter (tenacity) on transient failures only, permanent errors (bad host, oversized, disallowed MIME, 4xx) never retried.

Adapter framework (adapters/, ADP-001..007): the SourceAdapter Protocol exactly as

specified, plus one concrete, generic, non-source-specific implementation — DirectDocumentAdapter (fetches one configured literal URL). No CMS/Medicaid business logic; that's M2.

Ingest orchestration (ingest.py): run_discovery() — the one function both the manual-trigger

API route and the scheduler call, so their behavior can't drift apart. Implements ADP-004 idempotent rediscovery (identical bytes never create a duplicate DocumentAsset) without losing observation history (ING-007 — a new DocumentObservation row is still created every run), and ING-006 (permanent errors route the source to REVIEW_REQUIRED rather than retrying forever).

Scheduler (workers/loop.py, SRC-003): polls enabled sources whose cadence has elapsed, runs

discovery through the same code path as a manual trigger. Idles safely with zero sources registered.

Minimal internal API (api/): GET /healthz, /readyz, /v1/build (public); admin-token-gated

GET/POST /v1/sources, GET /v1/sources/{id}, GET /v1/sources/{id}/status, POST /v1/sources/{id}/discovery-runs. No public/customer-facing data of any kind ships this pass.

Deploy wiring: new {$POLICY_DOMAIN} Caddy block (mirrors verify-web's exact shape); new

postgres-init-policy/policy-migrate/policy-web/policy-worker services in deploy/ionos/docker-compose.yml (mirrors the verify service trio and the proven multi-database-on-one-shared-Postgres pattern); deploy.sh now health-waits on policy-web after the main restart and extends the existing alembic-revision-id-length guard (the 0026 incident's failure mode) to cover policy/'s migrations too.

Verified live in the real Docker image (not just unit tests)

Built the actual policy/Dockerfile image, ran policy-migrate's entrypoint for real (Alembic upgrade head succeeded, created all 5 tables), then ran policy-web and, against a real HTTP fixture server reachable from inside the container:

registered a source via the real API,
triggered a real discovery run — fetched, hashed, and archived a real document,
confirmed the archived bytes on disk match the source exactly (M1's stated exit criterion),
re-triggered discovery and confirmed no duplicate asset was created (idempotency),
confirmed the admin-token gate correctly rejects missing/wrong tokens and accepts the right one.

Verification

PYTHONPATH=policy/src:policy/tests python -m pytest policy/tests -q — 27 passed (adapter-fixture

fetch tests including SSRF-blocking and redirect-hop re-validation, storage idempotency, migration upgrade/downgrade/upgrade round-trip, full API contract tests).

python -m pytest tests/ -q — 334 passed (was 330; +4 new deploy-wiring regression tests).
PYTHONPATH=verify/src:verify/tests python -m pytest verify/tests -q — 824 passed, confirming

zero changes to verify/ (one transient failure seen once, reproduced as unrelated pre-existing flakiness on isolated re-run).

docker compose --env-file deploy/ionos/prod.env.example -f deploy/ionos/docker-compose.yml config

— confirms the new services resolve, dependency ordering is correct (postgres → postgres-init-policy → policy-migrate → policy-web/policy-worker → caddy).

bash -n deploy/ionos/deploy.sh — confirms the new deploy steps didn't break shell syntax.

Not done / explicitly out of scope

No CMS/Medicaid/MAC adapter business logic (M2). No policy identity/versioning/sectionization (M3). No diff engine, LLM extraction, evidence/confidence scoring, human review console (M4-M8). No customer REST API, entitlements, billing, webhooks, email/CSV, or MCP tools (M9-M10) — all gated behind the M0 legal/source-terms/licensing review, which has not happened. Everything shipped this pass is admin-token-gated internal infrastructure with zero sources registered by default.