Operational monitoring and alert hooks
Scrape metrics, inspect durable failures and deliver optional webhook alerts from the agent and gateway.
Initialize and run
Stop workers and run coldctl init for schema 17. The agent records failure/recovery transitions transactionally with job, queue and recurring-schedule changes, even when the monitor is stopped. Run the monitor as a separate service alongside the scheduler with the same --data-dir. Set the referenced secrets through your service manager. Omit webhook flags to disable delivery.
coldctl init
coldctl agent monitor serve --listen 127.0.0.1:9464 --token-env COLDCTL_METRICS_TOKEN --webhook-url-env COLDCTL_ALERT_URL --webhook-token-env COLDCTL_ALERT_TOKENAgent endpoints
GET /metrics exposes Prometheus text: job/queue/schedule counts, archive/deletion aggregates, queue age, sample freshness, active incidents and alert backlog. GET /failures returns the latest 100 transitions. GET /ready returns 503 for failed or stale samples; it reports monitor readiness, not worker liveness or archive correctness. The metrics token needs 32–4096 bytes. All endpoints require Authorization: Bearer TOKEN when configured; non-loopback binding requires a token. Use TLS at ingress. --poll-seconds defaults to 10, with a range of 1–300.
coldctl agent monitor statusGateway endpoints
The existing administrator bearer credential protects GET /metrics and GET /api/admin/monitoring/failures. Tenant users are denied. Metrics include durable HTTP outcome counters and available query workers. Incidents cover server errors, query/record/download execution rejections (including invalid filters and resource limits), failed catalog refreshes and background export failures. Operations resolve on their next success; this does not imply every prior request is recoverable. Streaming errors after headers are outside response outcome counters.
[alerts]
webhook_url_env = "COLDCTL_ALERT_URL"
webhook_token_env = "COLDCTL_ALERT_TOKEN"Webhook contract and privacy
The generic JSON event has id, service_id, component, kind, entity_id, state (firing/resolved) and occurred_at (Unix seconds). service_id is stable in the monitoring sidecar; map it to your deployment in the receiver. Agent entity IDs are internal UUIDs; gateway incidents aggregate per operation. No names, query values, row data, secret URLs or raw error text are sent. Metric labels are bounded and exclude resource IDs. Provider-specific Slack/PagerDuty formats require an adapter; hooks do not execute shell commands.
Durable delivery
Only 2xx acknowledges delivery. HTTPS is required, redirects are disabled, and requests time out after 10 seconds. --allow-loopback-http (gateway allow_loopback_http) permits only numeric loopback HTTP for tests. Retry starts at 30 seconds and doubles to one hour. At-least-once delivery uses Idempotency-Key equal to the stable event UUID; the receiver must deduplicate. Recovery cannot overtake undelivered firing for the same incident. Events observed with hooks disabled are not replayed when hooks are enabled; previously pending events resume. Changing the receiver redirects pending deliveries to its new URL.
Retention and backups
The agent source journal retains 10,000 transitions. The separate monitoring.sqlite sidecar retains delivered/disabled history and resolved incidents for 30 days; active incidents and pending alerts are retained. Stop its owning process before copying the sidecar. coldctl state backup includes the source journal, not monitoring delivery acknowledgements. Restore matching backups; a regressed source journal makes monitoring unready rather than silently skipping new events. Review pending alerts before replacing mismatched sidecar state.
Outage detection and limits
coldctl_agent_journal_gap and the durable coldctl_monitor_journal_gaps_total expose journal overflow after extended monitor downtime. Current failure states are reconciled after catch-up; missing history cannot be reconstructed. Use external scrape/process alerts for stopped workers/monitors, stale samples, increasing oldest-due queue age, disk pressure and pending webhook age. A dead process cannot notify its own outage. Monitoring does not restart workers, resume archives or sync Cloud metadata. Gateway delivery runs during serve; export jobs remain in memory.