Production
performance.
Where the experience slowed.
What the evidence says. What to check next.
- Booking reads over 30s
- 48.96% 2,707 / 5,529 logged requests
- Website-worker heap fatal
- 1 record Confirmed at 13:04:57 IST
- Connection-acquisition waits
- 28 records 12 API + 16 tasks · cause unproven
Booking reads slowed sharply.
GET /booking-enquiriesAn existing tail worsened—not a new slow route. Prior-day comparison is not a matched weekday; traffic mix can differ.
Four lenses. No single verdict.
Latency, process health and completed work answer different questions.
Different signals. Different times.
07 Sep · all times ISTContent memory pressure
24 one-minute Maximum bins above 90%.
Content task gap
Zero running tasks at a sample; desired count 1.
Website heap OOM
JavaScript heap fatal confirmed; not a container-kill finding.
Booking read tail emerges
No >30s records earlier that day; sustained late-day tail.
API error burst
35 then 33 target 5xx. Fast failures, not the slow-success tail.
Tasks pool waits
16 connection-acquisition timeout log records.
Ordered observations, not a causal chain. Spacing is categorical, not proportional to elapsed time.
Follow the evidence to a handoff.
Proposed daily practice for on-call and service owners. No job, alarm or schedule is installed.
- 01
Bound
Complete IST day
Prove coverage - 02
Compare
Same route + method
Rank latency and errors - 03
Correlate
Same time + service
Check process and outcomes - 04
Record
Evidence + owner
Set the next review
Evidence & operating notes.
Telemetry: 7 Sep 2026 IST. Evidence reviewed: 8 Sep. A new page build does not refresh these measurements.
Review priorities · full findings
Priority matrix
These are review priorities for the historical findings, not declarations of an active incident. Confirm fresh recurrence and user impact before assigning incident severity. Owner lanes are suggested roles, not assigned individuals.
| Review lane | Confirmed September 7 signal | Next evidence / suggested owner |
|---|---|---|
| First · request latency | Booking-enquiries dominated the severe tail; no >30s records before 16:00 IST, then sustained late-day slowness. Inbox GET had 6/220 >30s and a 161.593s maximum, all non-5xx. | API owner: inspect dependency timings for the same service, instance and time; investigate repeated ~44–45s waits without assuming a timeout mechanism. |
| First · worker continuity | Website heap OOM; content memory >90% in 24 consecutive one-minute Maximum bins, then a zero-task sample while desired count stayed 1. | Platform + worker owners: obtain stopped-task reasons, heap settings, job type and queue age. Do not label content-worker OOMKilled without evidence. |
| First · connection access | 28 pool-acquisition records. Tasks cluster at 19:46–19:48 IST; API records occur in separate windows. | DB + service owners: correlate pool used/free/pending, long transactions and lock waits if available. Do not attribute all slow routes to these records. |
| Next · API failures | POST /requests-api/filters: 59/154 5xx. Separate 16:23–16:24 IST burst: 35 + 33 target 5xx; these failures were fast. | API owner: separate exception classes and request context from the slow-success investigation. |
| Next · conversational AI | Frontdesk: 25/658 empty-output attempts vs 21/579 prior; 24 traces, 16 failure-only in the observed window. | Frontdesk owner: inspect authorized conversation outcomes; empty output alone does not prove caller-visible failure. |
| Next · integration correctness | GBP createPost: 928/1,795 failed attempts; updatePlaceActionLink: 57/62. Similar failures existed the prior day. | Integration owner: group canonical request outcomes by operation and status; validate resources/inputs before retrying. No customer identities belong in this page. |
| Planned · chronic latency | Conversation-partners median 11.39s vs 11.22s prior; 6,255/7,129 >5s. Layout DELETE median 13.16s, improved from 15.67s but still slow. | Route owners: distinguish workload growth from percentile regression; establish interactive vs background expectations. |
| Planned · AI waste and delivery | Scraper: 98 incompatible-temperature failures, all 49 traces also contain success. Blog-content p95 82.72s vs 67.55s; all successful worker work. | AI + content owners: verify caller/model compatibility and completed jobs; review job completion latency, not just provider-call duration. |
Read-only procedure · five checks
Run the review
Pin the window and prove coverage
Use inclusive start and exclusive end. September 7 IST is [2026-09-06T18:30:00Z, 2026-09-07T18:30:00Z); the prior IST day starts 2026-09-05T18:30:00Z. Apply date-partition pruning and actual UTC timestamp bounds to Athena. CloudWatch Logs queries also need explicit exclusive-end guards.
Save query IDs, UTC bounds, timezone, region, source, scan cost and result completeness. Check first/last timestamps, missing fields, exact duplicates and missing service-hour buckets. An empty series means unknown, not zero. Stop short of a “healthy” conclusion when access or ingestion coverage is missing.
Rank HTTP symptoms—not the platform average
Group by service + method + normalized route, removing query strings and identifiers. Report count, >2s / >5s / >10s / >30s counts and rates, p50/p95/p99/max, 5xx and 429. Keep method and service distinct. Split OPTIONS, interactive routes, webhooks and background traffic.
Compare identical windows and same-hour patterns. Calculate approximate percentiles from raw rows; never average route or interval percentiles. Retain a few sanitized anomaly lookup keys privately: UTC timestamp + service + instance/stream + request ID, and trace ID when present. Bare request IDs are reused.
Cross-check infrastructure and process evidence
For affected ECS services, inspect CPU and memory Average/Maximum, running vs desired tasks, and raw process logs. Use five-minute metrics for the sweep, one-minute metrics around anomalies. Search exact technical signatures for pool-acquisition timeouts and JavaScript heap OOM; broad words like “timeout” can occur in payloads.
For ALB, separate target 5xx from ELB-originated 5xx; inspect request count, target response time and unhealthy hosts. For Aurora, inspect ACU utilization/capacity, CPU, connections, DB load and deadlocks. For Redis, inspect engine CPU, memory, evictions, rejected connections and replication lag. Add lock/pool detail only when that evidence is accessible; never print SQL bindings or customer payloads.
Review AI and integration outcomes
For LLM events, group feature + service + provider + model + status. Report attempts, failures, latency tails and error-signature families. Split interactive-adjacent work from worker throughput. Follow sampled traces to distinguish retries/fallbacks from final business outcomes; do not count attempts as jobs or users.
For GBP, use canonical request events and metadata outcomes/durations, not wrapper error-log counts. Group by operation and upstream status. Review repeated resource/input failures and integration-recovery skips separately from latency. Restrict customer-specific follow-up to authorized internal tools.
Reconcile alarms, then write the handoff
Read historical state reasons, dimensions, statistic, period and evaluation window. State transitions are not incidents, and scale-in/security alarms are not automatically performance failures. Compare actual metrics/logs with the evaluated samples; alarm delivery time is not metric sample-start time.
Write confirmed symptom → impact evidence → hypothesis → next check. Assign a human owner and next review time. Close only after comparable-window evidence shows recovery, with business completion checked where relevant. Remediation and production changes require their normal authorization; this runbook is read-only.
Proposed triggers & historical alarms
Investigation triggers vs existing alarms
Proposed manual investigation triggers
These are starting points for owner review, not configured alerts, contractual SLOs or an approved paging policy. Check sample size and workload class before escalating; no-data results always need a coverage investigation.
| Signal | Proposed trigger | What to do next |
|---|---|---|
| Interactive route latency | Any observed >30s response; or p95 >5s with at least 100 records in a 15-minute window | Inspect samples immediately; compare same-route baseline and user-facing completion. A single sample warrants review, not automatic incident paging. |
| Route errors | At least 5 server errors and >5% 5xx in a 15-minute window | Group exceptions and status codes; compare edge and application evidence. |
| Worker continuity | Any heap-fatal record; or a zero-running-task sample when desired >0 | Verify process/task lifecycle and queue age; establish whether work stopped and for how long. |
| Memory / pool waits | Memory Maximum >90% for 3 consecutive one-minute bins; or any exact pool-acquisition timeout | Inspect averages and per-task detail. Threshold crossings are not proof of a leak, saturation or lock cause. |
| AI / integrations | Any new configuration-error signature or empty-output failure; repeated canonical operation failures | Review recovery and final outcomes; prioritize user-blocking failures over successful background tails. |
Historically verified alarm evaluations
The September 7 alarm cross-check verified these evaluation thresholds from historical state-reason data. Re-read current alarm configuration before relying on them today; this is not a complete alert inventory.
| Alarm family | Verified historical evaluation | Interpretation guardrail |
|---|---|---|
| website-worker CPU | Five-minute Average against 85% | Historical windows differ from aligned fixed buckets. |
| content-worker memory | Five-minute Average against 90% | Service memory is not the JavaScript heap limit. |
| API target 5xx | 10 target errors/minute threshold; 35 and 33 in 16:23/16:24 samples | Distinct from ALB-originated errors; alarm entered at 16:27:07 IST. |
| Aurora WriteIOPS | One-minute Average against 5,000 ops/s | IOPS alone does not prove disk saturation. |
| Aurora connection anomaly | Anomaly-model upper-bound breaches | Not a configured max_connections exhaustion signal. |
There were 1,581 state transitions and 24 alarm names entering ALARM in the inspected region/window—not 24 incidents. Repeated response-time transitions are a reason to review noise and coverage, not a substitute for raw latency evidence.
Coverage gaps & interpretation guardrails
Keep the evidence honest
The sweep confirms HTTP tail latency, pool waits, a website-worker heap OOM and content-worker memory pressure/task unavailability at a sample. It does not establish one shared cause. The heap error, later CPU peak, API burst and late-day request slowdown have different timelines.
- Do not use the global median as UX. The 723,590 target HTTP records include 124,770 fast OPTIONS records and internal/automated traffic. Only 63 HTTP records have trace IDs. Missing joins are an observability gap, not evidence of no relationship.
- Keep denominators separate. HTTP logged requests, ALB-routed requests, log records, AI attempts and business jobs are different populations. HTTP reported 187 5xx; ALB target metrics reported 186. Do not force them to agree.
- Read peaks correctly. A peak five-minute p95 is not a daily p95. Maximum-bin breaches do not prove the entire bin was above threshold. A zero-task sample does not determine exact outage duration.
- No-data is not health. Aurora PostgreSQL logs were unavailable in-window; some AWS topology APIs were denied. Stopped-task reasons and deployed revision were not established. Zero measured deadlocks does not exclude lock waits.
- Avoid false recovery claims. Scraper fallback success is LLM-stage evidence, not proof that every scrape completed. Frontdesk empty output needs conversation review. Repeated GBP failures are attempts, not distinct posts lost.
- Keep comparison fair. The prior day is not a matched weekday. Traffic mix changed; conversation-partners got more traffic without a broad percentile regression. Use exact counts/maxima alongside sample-sensitive percentiles.
Handoff template · record the next review
Record a review
Copy this template into the team's approved incident/review record. Preserve private query artifacts in restricted storage; publish only sanitized aggregate findings. Update this page's baseline and freshness labels manually only after a new verified sweep.
Review date / operator / next review:
Scope: environment, region, exact UTC [start,end), display timezone
Comparison window / deployed revision (or unknown):
Coverage: sources, row counts, missing series, denied access, late arrivals
Symptom: service + method + normalized route or feature / operation
Observed: count, denominator, rate, p50/p95/p99/max, interval/statistic
Business impact: verified outcome / not yet established
Confirmed evidence: query IDs, metric dimensions, restricted artifact location
Hypothesis: explicitly unproven; supporting and contradicting evidence
Action: next read-only check, human owner, review deadline
Closure: comparable-window recovery and completion evidence / still open
Sources, query handles & repository context
Sources and freshness
Telemetry baseline: September 7, 2026 IST, compared with September 6 where available. Evidence reviewed: September 8, 2026. The page is manually maintained; a newer page build does not refresh the measurements.
Source snapshot: repository zoca-ai/mononest, previously recorded origin/develop snapshot, SHA aae2d9f3022d649cbc8e64c3d7c2ef17c98c53cf. Retained from the earlier evidence review; no current-code contract is asserted by this visual redesign. This records the documentation's repository context, not the production revision deployed during the sweep. This is an operational how-to backed by telemetry reports, not an implementation or root-cause audit; no application tests were run and no deployed-code attribution is claimed.
Evidence ledger · sanitized provenance and limitations
| Evidence | Primary source / reproducibility handle |
|---|---|
| HTTP performance report | Athena zoca_logs_production.http; aggregate query 54ff1bf5-c811-4d5e-be5b-cc6fd2be0129; source report perf-sweep-20260907-http/findings.md |
| Infrastructure report | CloudWatch metrics + ECS Logs Insights; exact-signature query 36e65a45-ad6c-44fa-a963-769e9e6b53e1; perf-sweep-20260907-infra/FINDINGS.md |
| LLM performance report | Athena zoca_logs_production.llm; aggregate query d1190e34-19e9-40d8-9f09-f9050cfeefe9; perf-sweep-20260907-llm/findings.md |
| App / GBP report | Canonical request query c870988a-2cb3-4bd8-ad28-8c809c96f1b0; perf-sweep-20260907-app/findings.md |
| Alarm cross-check | Complete paginated state history and historical reasons; perf-sweep-20260907-app/alarms_findings.md |
Original reports and query artifacts were supplied in local restricted working directories under /tmp for this review; they are not hosted by this docs page and are not a durable shared archive. The review owner should retain them in approved restricted storage before local cleanup. Raw reports may contain sensitive investigation context and must not be uploaded wholesale. Query IDs support investigation but do not guarantee indefinite result retention.
HTTP/LLM telemetry represents completed, logged operations, not all attempts or browser end-to-end latency. No late-arrival audit proves complete ingestion. Infrastructure covered discovered metric dimensions, not every resource in the account. No broad interactive LLM latency incident or vendor-wide GBP outage was demonstrated.