Skip to main content

​
Metrics, dashboards & alerts

The FERAL Brain exposes a Prometheus-compatible /metrics endpoint backed by a named registry in feral-core/observability/metrics.py. A default Grafana dashboard and Prometheus alert rules ship in-tree under ops/.

​
Endpoint

GET /metrics returns text in the standard Prometheus 0.0.4 exposition format. Two environment switches gate it: The off-loopback default returns 404 (not 401/403) so the response is indistinguishable from “endpoint not mounted” for unconfigured installs.

​
Registered metrics

Every metric below is defined in feral-core/observability/metrics.py and referenced by either the bundled dashboard, the alert rules, or both. The tests/test_metrics_registry.py suite enforces that contract. Some emitters are still being wired into all call sites; the registry surface and the underlying metrics are stable. Missing emit() calls show up as zero-valued series, not as registry errors.

​
Retention

/metrics exposes the current in-process state — the Brain holds no historical samples. Persistence is the scrape pipeline’s job:
  • Recommended retention: 30 days at 15-second scrape interval (the default Prometheus retention is 15 days; double it for the canary-grade signal the alert rules assume).
  • Cardinality budget: every feral_http_requests_total series is bounded by method × route_template × status_class. Routes are recorded using FastAPI’s path template (e.g. /api/jobs/{id}), not the raw URL, so cardinality stays in the low hundreds even under heavy traffic.

​
Importing the dashboard

The bundled dashboard lives at ops/grafana/feral-overview.json (Grafana schema v39, validated against Grafana 11+).
The dashboard expects a Prometheus datasource — pick yours from the Datasource template variable at the top of the dashboard once it loads.

​
Alert rules

ops/prometheus/alerts.yml is loaded directly by Prometheus via the rule_files directive. Drop it next to your existing rule files and reload Prometheus (SIGHUP or POST /-/reload).

​
Runbooks

​
HighErrorRate

  1. Check journalctl -u feral (or your equivalent log aggregator) for tracebacks.
  2. curl -fsSL http://localhost:9090/health to confirm the Brain is up.
  3. If the spike correlates with a deploy, roll back via pip install 'feral-ai==<previous-version>' and restart.

​
LLMAllProvidersDown

  1. feral doctor to see which providers errored and which keys are configured.
  2. Validate API keys via feral key list and probe each provider in Settings → Providers.
  3. If a provider is genuinely down, the chain will recover automatically once it comes back; the alert auto-resolves.

​
SyncPeerDown

  1. feral sync peers to list discovered peers.
  2. Check mDNS / passphrase / firewall on both ends.
  3. If the deployment is intentionally single-node, set FERAL_SYNC_DISABLED=1 and remove the feral_sync_was_active_recent gauge from being set to 1 — the alert will then stay quiet.

​
SupervisorBacklog

  1. Drain the queue from the WebUI Supervisor → Approvals page.
  2. If approvals are blocked because the deciding human is offline, increase the alert threshold or page the on-call.

​
VaultDecryptFailed

  1. Treat as a security-relevant signal: AEAD failures should never happen in steady state.
  2. Confirm the OS keychain still holds the master key (feral key status).
  3. Inspect feral-core/security/vault.py logs for the failing entry.

​
Adding a new metric

  1. Define the metric in feral-core/observability/metrics.py against REGISTRY, register it in the _METRICS map, and write a one-line docstring naming the eventual emitter module.
  2. Add at least one panel in ops/grafana/feral-overview.json or an alert rule in ops/prometheus/alerts.yml that consumes it. The test_metrics_registry.py::test_no_orphan_metrics test fails the build otherwise.
  3. Add an emit() call site inside the owning module. Keep it cheap — the helper no-ops when the kill switch is off, but the label dict still allocates.