A report that arrives in useful layers
Start one durable analysis, show honest progress, publish a validated baseline, then add modeled insights—without making the browser responsible for the job.
The experience
Stream progress and validated sections, not unfinished money. This proposal is for engineering and product review. It describes intended behavior, not an available API or a shipping feature.
| What the user sees | Meaning |
|---|---|
| Analysis accepted | A durable run exists; paid work may not have begun. |
| Preparing keywords | Candidates, buckets, prices and scan geometry are being prepared and validated. |
| Measuring local visibility | Completed point outcomes and coverage advance against a frozen scan plan. Results remain provisional. |
| Baseline ready | A deterministic gate approved an immutable estimate based on demand, prices and observed ranks. |
| Adding modeled insights | Competitor profiles and eligible model features are being resolved. The baseline remains readable. |
| Complete, partial or blocked | The UI explains what was published, what is missing and whether recovery is possible. |
A disconnected browser does not cancel the job. Reopening the report restores persisted state. No monetary token streaming: do not progressively assemble revenue digits, expose unvalidated model output, or imply that an incomplete subtotal is a full estimate.
Request and observation
Scroll horizontally to follow all four lanes →
The command and observation paths have separate responsibilities:
- Start: a POST with an idempotency key checks current location access and business ownership, then atomically creates the LOCATION
KI_RUNroot inkeyword_pipeline_execution(KPE), the input snapshot and request identity. Enqueue only after commit. A repeated equivalent command resolves to the same run; the conflicting-payload response and key lifetime remain decisions. - Observe: a separate SSE GET authenticates the reader and reads committed run state. It neither starts nor retries paid work.
process_id = execution_id = keyword_rank.job_id; APIrunIdis the alias. - Execute: workers acquire a fenced lease. Every conditional write checks owner, process and lease token; a superseded worker cannot publish. Resolve ownership through the root/input, not an arbitrary caller-supplied place ID.
Commit-to-queue delivery still needs a recovery mechanism. Enqueue-after-commit avoids processing uncommitted input but does not close the crash window. Decide between a transactional outbox and an idempotent sweeper/reconciler before implementation.
Gated generation and baseline
Scroll horizontally to follow all four lanes →
Prepare in parallel, join deliberately
Generate candidate keywords and run a separate evidence-bound input LLM validation. Independently prepare business buckets, resolve website-service prices and freeze the 25-point geometry. Preserve long-tail, adjective, synonym and trend provenance without treating subjective search terms as verified business claims.
Only input PASS releases accepted seed input to Google Ads. Persist demand, deduplicate canonical groups and map them to process-owned buckets. Only relevance PASS releases accepted groups for frozen selection and Maps. REJECT items are excluded and counted; REVIEW items stay held. Unavailable mandatory validation blocks downstream work—it is not permission to skip a gate.
Selection proposes known demand ≥20, using fiveMilesSearch under existingLocalDemand-v1. This local metric is a recommendation from the TRD, not a ratified choice; raw Ads volume versus local demand is still open. Pin the chosen metric, threshold, cap and selection policy with the run. New seeds after selection freezes require a new process.
Record outcomes, not guesses
Scan each selected keyword at the exact 25 frozen coordinate pairs: the center plus eight bearings on each of the 1-, 3- and 5-mile rings. A fenced transaction locks the point execution, stores validated rank rows and the explicit query outcome atomically, and marks the batch complete. Re-observing a completed batch must not insert it again.
Keep rank 20, valid absence, successful empty response, failed query, invalid location and pending distinct. Missing rank rows are never evidence of absence. Progress counters come from persisted outcomes against the frozen keyword × point plan, not the number of rows returned. Show planned, successful, failed and pending coverage; label intermediate visibility provisional. Failed points never donate their weights to successful points.
Publish only after deterministic validation
Visibility includes coverage. Baseline revenue depends on demand, frozen prices and observed ranks, not competitor enrichment or model inference. Prices come from eligible stored website-service data with service, variation, currency and source lineage; no automatic competitor, GBP or AI price fallback. LLM-origin discoveries need owner confirmation or independent existing-source verification before monetary use.
Allocate canonical demand once across applicable offered services, including unpriced services. Shares sum to one; do not redistribute unknown shares. A known subtotal may be shown with coverage and a null full total. Missing, range, “from”, unit and variable prices remain unavailable pending policy.
A deterministic gate recomputes values and checks conservation, decimal rounding, currency/version alignment, nulls and coverage. Publish the immutable revision and its current-report pointer atomically only on PASS. Quarantine failed revisions; preserve the previous validated report. Estimates are not actual sales or guaranteed lift.
Inference and recovery
Scroll horizontally to follow all four lanes →
Build the union of competitor place IDs returned across scans and enrich full business profiles in parallel with the baseline branch. Model inference waits for eligible, time-aligned features and an approved, pinned model pack, with feature/provider/market parity checks. Record pack hash and feature/scenario versions. Keep observed-baseline gapfix-v1 and signed model-only growth13-v1 separate.
Validate modeled final output deterministically before atomically publishing another immutable report revision. Inference or profile failure must not invalidate a validated baseline. Do not silently substitute invented features or a different model pack.
Daily training is independent. SYSTEM execution state tracks training and model promotion; it does not block a location run, trigger a daily business rescan or change an existing run's pinned model. Failed promotion leaves the compliant champion unchanged.
Proposed event contract
These event names and payload responsibilities are proposed, not an implemented wire contract. Delivery is a hint to refresh persisted truth, not the source of truth. Repeated notifications must be safe.
| Event | Intended client meaning |
|---|---|
run.accepted | The durable run is accepted; identify the run and current state revision. |
stage.updated | A persisted stage changed; include status, gate verdict, coverage or reason as applicable. |
scan.progress | Persisted completed/failed/pending counts against the frozen plan; explicitly provisional. |
section.ready | A validated non-monetary section is readable at an identified revision; not a promise of a full report. |
report.published | An immutable baseline or final report revision passed its gate and became readable. |
run.blocked | A required dependency, access check, validation or reconciliation prevents progress; disclose the reason. |
run.partial | Work ended with an explicitly permitted incomplete scope; identify omissions and the last valid report, if any. |
run.completed | Required work for the agreed completion policy is finished; identify the latest published report. |
All notifications should identify runId, a monotonic stateRevision, schema version and committed state. Report notifications additionally identify reportRevision and kind. Keep observed scan counters separate from publication status. A section event never bypasses the revenue gate. Exact reason codes, envelope fields, heartbeat cadence and terminal-state transitions need ratification.
Illustrative API shape — not implemented endpoints
The paths and JSON below explain the split; they do not document deployed controllers. Placeholder values are illustrative, not a captured production response.
POST /keyword-intelligence/runs
Idempotency-Key: <client-generated-key>
Content-Type: application/json
{"locationId":"<authorized-location-id>"}
202 Accepted
{"runId":"<run-id>","eventsUrl":"/keyword-intelligence/runs/<run-id>/events"}
GET /keyword-intelligence/runs/<run-id>/events
Accept: text/event-stream
event: report.published
data: {"schemaVersion":1,"runId":"<run-id>","stateRevision":42,"reportRevision":1,"kind":"baseline"}
The numeric revisions are illustrative. An implementation must define the snapshot transport and atomic revision allocation before clients rely on this envelope.
Reconnect from a snapshot
KPE is a state store, not a durable event log. Recommend snapshot recovery rather than claiming Last-Event-ID replay. A monotonic state/report revision cursor is a freshness check, not proof that every historical event can be replayed.
- On every connect or reconnect, reauthenticate and recheck current location access plus the frozen owner relationship. Fail closed if ownership changed.
- Read a consistent snapshot containing run/stage states, gate receipts, scan coverage and the latest published report reference, with state and report revisions.
- Replace client state with the snapshot; do not merge stale counters or append old sections. Then apply only newer updates and never roll back the visible report revision.
- Close the snapshot/subscription race: subscribe before reading and discard buffered revisions at or below the snapshot, or use a revision-aware catch-up read. Exact server strategy is pending.
- If notifications are missed, duplicated or delivered out of order, reread current state. Reconnecting, refreshing and reading a report never initiate paid work.
Commit state before notification. A lost notification must not lose the result. Durable full replay would require an explicitly designed retained event log and replay/retention semantics; this proposal does not assume one exists.
Failure boundaries
| Failure or incomplete scope | Safe proposed behavior |
|---|---|
| Mandatory input/relevance gate unavailable | Block Ads/selection/Maps at the corresponding boundary; retain evidence and explain why. |
| Ordinary generation chunks or submitted Ads requests unfinished | Do not declare complete discovery. Terminal failure policy must be ratified before allowing a partial baseline. |
| Grounded trend discovery fails terminally | May continue only as explicitly disclosed partial discovery under an approved policy; never silently call it complete. |
| Scan points fail | Preserve explicit outcomes and coverage. Do not infer absence or reweight; partial-baseline eligibility still requires approved terminal-failure policy. |
| Provider outcome unknown but may have been charged | Pause for reconciliation; do not blindly retry and risk duplicate spend. |
| Revenue gate fails | Quarantine candidate revision; keep the last validated report readable. |
| Competitor profile or inference fails | Preserve baseline; explain unavailable modeled insights and allow only policy-approved recovery. |
| Worker lease expires or queue notification is lost | Fence stale writes; recover through durable state and approved retry/enqueue reconciliation. |
| SSE disconnects | Job continues; recover via authorized snapshot. |
A mathematically truthful partial estimate is not automatically permission to publish after any terminal failure. The deterministic revenue gate and the operational eligibility policy are separate requirements. Recovery of failed points can publish a new validated revision with pinned inputs; completed successful outputs remain stable. New inputs or a fresh scan require a new process.
Decisions before implementation
- Demand and selection: raw versus local threshold metric, keyword cap, freeze rules and the UI label for the chosen metric.
- Partial completion: which terminal generation/Ads/scan failures permit a baseline, minimum coverage, reason codes and how blocked, partial and completed differ.
- Exactly-once intent, at-least-once delivery: idempotency scope, payload conflicts, expiry, enqueue crash-window recovery, lease renewal and reconciliation authority.
- Transport: snapshot endpoint or initial SSE message, consistent snapshot/subscription handoff, cursor allocation, reconnect backoff, heartbeat and polling fallback. Add durable replay only with an explicit log design.
- Economics and models: supported markets/currencies, booking assumptions, variable-price treatment, provider quota/cost gates, approved pack eligibility and benchmark/promotion criteria.
- Retention and security: evidence/report lifetime, SSE connection limits, access revocation during a stream, role/privacy rules and legacy-reader isolation before KI writes.
Review checklist
Validate idempotent starts and payload conflicts; enqueue-after-commit crashes; stale-worker fencing; all three gates and unavailable validators; demand 19/20; exact frozen geometry; atomic point retries; rank 20 versus absence/empty/failure; null totals and unpriced shares; competitor union; inference failure preserving baseline; immutable report revisions; lost/duplicate notifications; reconnect during publication; owner changes; and independent failed-training continuity.
Source and design boundary
Primary source: user-supplied Keyword Intelligence — TRD, dated 15 October 2026, file doc_f4379fb508f2_keyword-intelligence-trd.html. The source explicitly calls itself a proposed technical design and says no application code or migration was executed. This page does not claim develop-branch implementation evidence.
Inherited from that TRD: reuse-first KPE state, three blocking gates, fenced leases and retries, demand/rank reuse, frozen 25-point scans, explicit outcomes, pricing/coverage rules, immutable validated revenue revisions, baseline independence, pinned-model inference and independent training.
Streaming design additions on this page: the command/observation split, proposed event vocabulary and envelope, snapshot-replacement reconnect semantics, revision cursors, commit-before-notify, queue crash-window and subscription-race recommendations, and the user-visible progression. These are recommendations for ratification, not claims that the TRD already specifies SSE or that code implements it.
The supplied HTML's related contract, prompt, SQL and technical-reference files were not supplied with this request. They are intentionally not linked here. Review the pending decisions above before treating any illustrative API shape as a delivery contract.