> ## Documentation Index > Fetch the complete documentation index at: https://firebolt-aggregate-helm-docs-pr-38.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring > Prometheus metrics exposed by the Firebolt Operator and managed components. This document describes how the Firebolt Operator exposes Prometheus metrics for the components it manages. ## Metrics endpoints | Component | Port | Name | Path | What it exposes | | --------------------- | -------------------------------------- | ----------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Engine pods | 9090 | `metrics` | `/metrics` | `firebolt_running_queries`, `firebolt_suspended_queries`, and other engine-internal gauges. The Firebolt Operator scrapes the first two via `Pods/proxy` to drive both the drain check and auto-stop. | | Gateway pods (Envoy) | 9090 (default) | `metrics` | `/stats/prometheus` | Envoy connection, request, and cluster stats | | Firebolt Operator pod | Configurable via `metrics.bindAddress` | `https` or `http` | `/metrics` | controller-runtime reconciliation, workqueue, REST client, and Go runtime metrics | The gateway metrics port defaults to 9090 and is configurable per FireboltInstance CR via `spec.gateway.metricsPort`. Metadata pods do not currently expose a Prometheus metrics endpoint. ### Firebolt Operator metrics mode The Firebolt Operator metrics endpoint mode is controlled by two Helm values: | Mode | `metrics.secure` | `metrics.bindAddress` | Port name | Scheme | | --------------- | ---------------- | --------------------- | --------- | -------------------------------------------- | | HTTPS (default) | `true` | `:8443` | `https` | `https` with authn/authz and self-signed TLS | | HTTP | `false` | `:8080` | `http` | plain `http` | The Firebolt Operator PodMonitor template automatically adapts its port reference, scheme, bearer token, and TLS configuration based on `metrics.secure`. ## Scraping with Prometheus The Firebolt Operator Helm chart ships optional `PodMonitor` resources (one per component type) that can be enabled via `values.yaml`: ```yaml theme={"theme":{"light":"css-variables","dark":"css-variables"}} podMonitor: engines: enabled: true gateway: enabled: true operator: enabled: true allNamespaces: false # set true when the Firebolt Operator watches all namespaces ``` Each PodMonitor uses label selectors to match the relevant pods: * **Engines**: `firebolt.io/engine` (exists). Matches all engine pods regardless of engine name * **Gateway**: `firebolt.io/component=gateway` * **Firebolt Operator**: `control-plane=controller-manager` + chart selector labels When `allNamespaces` is true, `namespaceSelector.any: true` is added so pods in any namespace are discovered. This does not apply to the Firebolt Operator PodMonitor because the Firebolt Operator always runs in the release namespace. ### Per-instance monitoring The chart-level PodMonitors apply uniform scrape configuration to all instances in scope. If you need per-instance control (different intervals, selective enablement, custom relabelings), disable the chart-level PodMonitors and deploy your own alongside each FireboltInstance or FireboltEngine CR. The label selectors to use are: * Engine pods: `firebolt.io/engine: ` * Gateway pods: `firebolt.io/instance: `, `firebolt.io/component: gateway` ## Architecture decisions ### Helm chart templates, not Firebolt Operator reconciliation PodMonitor resources are shipped as Helm templates, not created by the Firebolt Operator's Go reconciliation loop. This follows the dominant industry pattern used by cert-manager, Strimzi, FoundationDB operator, and others. CloudNativePG tried a reconciler-managed approach and deprecated it in v1.26 because: * It creates a hard dependency on the Prometheus Operator CRDs. The Firebolt Operator fails to reconcile on clusters where the CRDs are not installed. * The Firebolt Operator overwrites user customizations (scrape intervals, relabelings, TLS config) on every reconcile. * It adds RBAC complexity for `monitoring.coreos.com` resources. * Platform teams want full ownership of their monitoring configuration. ### Gateway stats listener The Envoy admin interface (port 9901) is bound to `127.0.0.1` and must stay that way. It exposes mutation endpoints (`POST /healthcheck/fail`, `POST /quitquitquit`) that the preStop hook depends on for graceful shutdown. Binding admin to `0.0.0.0` would allow any pod in the cluster to drain or kill gateway pods. Instead, a separate read-only stats listener is added on the metrics port (default 9090). This listener proxies only `/stats/prometheus` from the admin interface via an internal static cluster, exposing no mutation endpoints. ### Consistent metrics port Engine pods and gateway pods both expose Prometheus metrics on a container port named `metrics` (default 9090). The gateway override lives on the FireboltInstance at `spec.gateway.metricsPort`. The Firebolt Operator stamps the corresponding `metrics`-named port on the rendered Envoy container. Engine pods carry the port via the per-FireboltEngine wiring. PodMonitors can therefore always reference `port: metrics` without knowing the actual port number. The metadata pod does not currently expose a Prometheus endpoint, so no `metrics` port is stamped there. ### Cross-namespace support When the Firebolt Operator watches all namespaces (`watchNamespace` is empty), engine, gateway, and metadata pods may live in namespaces other than the Firebolt Operator's namespace. Setting `podMonitor.allNamespaces: true` adds `namespaceSelector.any: true` to the PodMonitors so Prometheus discovers pods across all namespaces. ### Embedded CR status metrics The Firebolt Operator embeds custom Prometheus metrics that expose the status of every FireboltEngine and FireboltInstance it manages. These are level-triggered gauges updated on every reconcile. There are no timers or persisted timestamps, which is consistent with the Firebolt Operator's level-driven reconciliation model. Duration and trend analysis are left to PromQL. This follows the pattern used by ArgoCD (`argocd_app_info`), Flux (`gotk_reconcile_condition`), cert-manager (`certmanager_certificate_ready_status`), and Crossplane (`crossplane_managed_resource_ready`). ## Operator metrics reference ### FireboltEngine metrics | Metric | Type | Labels | Updated | Description | | ------------------------------------------- | ------- | ---------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `firebolt_engine_status_phase` | Gauge | `namespace`, `name`, `instance`, `phase` | Every reconcile | StateSet-style: 1 for the current phase, 0 for all others. Phases: `stable`, `creating`, `switching`, `draining`, `cleaning`, `stopped`. | | `firebolt_engine_status_condition` | Gauge | `namespace`, `name`, `instance`, `type` | Every reconcile | 1 when the condition is True, 0 when False or Unknown. Types: `Ready`, `InstanceReady`. | | `firebolt_engine_spec_replicas` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Desired replica count from `spec.replicas`. | | `firebolt_engine_active_generation` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Generation number currently serving traffic. | | `firebolt_engine_pods_ready` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Number of ready pods in the active generation. | | `firebolt_engine_pods_total` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Total pods in the active generation (includes non-ready). | | `firebolt_engine_draining_generation` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Generation being drained, or -1 if no drain is in progress. | | `firebolt_engine_last_reconciled_timestamp` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Unix timestamp of the last successful reconcile. | | `firebolt_engine_drain_check_errors_total` | Counter | `namespace`, `name`, `instance` | On drain probe failure | Cumulative count of drain probe failures (pod unreachable, metrics missing). | ### FireboltInstance metrics | Metric | Type | Labels | Updated | Description | | --------------------------------------------- | ----- | ------------------------------------------ | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `firebolt_instance_status_phase` | Gauge | `namespace`, `name`, `phase` | Every reconcile | StateSet-style: 1 for the current phase, 0 for all others. Phases: `Provisioning`, `Ready`, `Degraded`, `Failed`. | | `firebolt_instance_status_condition` | Gauge | `namespace`, `name`, `type` | Every reconcile | 1 when the condition is True, 0 when False or Unknown. Types: `Ready`, `MetadataReady`, `GatewayReady`, `AuthReady`, `EngineTLSReady`, `GatewayTLSReady`. | | `firebolt_instance_info` | Gauge | `namespace`, `name`, `id`, `postgres_mode` | Every reconcile | Always 1. Carries static metadata: instance ID and postgres mode (`internal` or `external`). | | `firebolt_instance_last_reconciled_timestamp` | Gauge | `namespace`, `name` | Every reconcile | Unix timestamp of the last successful reconcile. | ### JWT signing-key rotation metrics Present on every FireboltInstance, and flat zero on one that does not use auth. | Metric | Type | Labels | Updated | Description | | -------------------------------------------------------- | ----- | ---------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `firebolt_instance_signing_key_generation` | Gauge | `namespace`, `name` | Every reconcile | Monotonic signing-key generation counter. Increments once per completed mint, so a rotation that ran stays visible afterwards. | | `firebolt_instance_signing_keys` | Gauge | `namespace`, `name`, `phase` | Every reconcile | Keys currently tracked, by phase (`Active`, `ValidationOnly`, `Removing`). Exactly one `Active`; a non-zero `ValidationOnly` or `Removing` means a rotation is in flight. | | `firebolt_instance_signing_key_rotation_pending_step` | Gauge | `namespace`, `name`, `step` | Every reconcile | 1 for the rotation step awaiting engine convergence, 0 for the others. All zero when nothing is waiting. Steps: `AwaitingPromotion`, `AwaitingRetireAnchor`, `AwaitingRemoval`. | | `firebolt_instance_signing_key_rotation_pending_seconds` | Gauge | `namespace`, `name` | Every reconcile | How long the pending step has been waiting, 0 when none is. Rotation gates park indefinitely by design, so **duration**, not existence, is what separates a normal engine roll from a stuck fleet. | | `firebolt_instance_signing_key_rotation_lagging_engines` | Gauge | `namespace`, `name` | Every reconcile | Engines whose observed auth hash has not converged for the pending step. The true total, not the truncated name list in `status.auth.laggingEngines`. | ### Label glossary | Label | Meaning | | --------------- | ------------------------------------------------------------------------ | | `namespace` | Kubernetes namespace of the CR | | `name` | Name of the FireboltEngine or FireboltInstance CR | | `instance` | Name of the parent FireboltInstance (from `spec.instanceRef` on engines) | | `phase` | Current lifecycle phase | | `type` | Condition type (e.g., `Ready`, `MetadataReady`) | | `step` | Signing-key rotation step awaiting engine convergence | | `id` | Stable instance ID (ULID) | | `postgres_mode` | `internal` (Firebolt Operator-managed) or `external` (user-provided) | ### Example PromQL queries Engine not ready: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_engine_status_condition{type="Ready"} == 0 ``` Engine stuck in draining phase for more than 10 minutes: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_engine_status_phase{phase="draining"} == 1 unless firebolt_engine_status_phase{phase="draining"} offset 10m == 0 ``` Scaling in progress (ready pods less than desired): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_engine_pods_ready < firebolt_engine_spec_replicas ``` Instance degraded: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_instance_status_phase{phase="Degraded"} == 1 ``` Stuck controller (no reconcile for 5 minutes): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} time() - firebolt_engine_last_reconciled_timestamp > 300 ``` Drain probe failures spiking: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} rate(firebolt_engine_drain_check_errors_total[5m]) > 0 ``` Fleet overview (ready engines per instance): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} count by (namespace, instance) (firebolt_engine_status_condition{type="Ready"} == 1) ``` Signing-key rotation stalled for more than 30 minutes: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_instance_signing_key_rotation_pending_seconds > 1800 ``` Which step, and how many engines it is waiting for: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_instance_signing_key_rotation_pending_step == 1 firebolt_instance_signing_key_rotation_lagging_engines > 0 ``` A rotation is in flight (more than one key tracked): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} sum by (namespace, name) (firebolt_instance_signing_keys) > 1 ``` ### Alerting on a stalled rotation Every irreversible rotation step waits for the whole engine fleet to converge, and waits indefinitely — parking is the safe behaviour, so a stuck rotation raises no error and leaves `AuthReady` `True`. That makes elapsed time the signal worth alerting on. A normal rotation parks each gate only for as long as one engine roll takes, so a threshold well above your rollout time separates the two: ```yaml theme={"theme":{"light":"css-variables","dark":"css-variables"}} - alert: FireboltSigningKeyRotationStalled expr: firebolt_instance_signing_key_rotation_pending_seconds > 1800 for: 10m labels: severity: warning annotations: summary: >- Signing-key rotation on {{ $labels.namespace }}/{{ $labels.name }} has not advanced for over 30 minutes. description: >- An engine has not converged on the current auth configuration. Check status.auth.pendingRotationStep and status.auth.laggingEngines on the FireboltInstance, then the named engines' own conditions. ``` The operator also emits a `SigningKeyRotationStalled` Warning Event once a step has been parked past its own threshold. The Event message is deliberately stable across reconciles so the API server aggregates the repeats into one Event with a count, rather than one Event per requeue. ### Cardinality Each FireboltEngine produces approximately 15 time series (6 phases + 2 conditions + 7 scalar gauges). Each FireboltInstance produces approximately 20 series (4 phases + 6 conditions + 1 info + 1 timestamp + 3 signing-key phases + 3 rotation steps + 3 scalar rotation gauges). For a cluster with 10 instances and 50 engines, expect roughly 950 series from the Firebolt Operator. This is negligible for any Prometheus deployment. The `phase` and `step` labels on the signing-key metrics are bounded enumerations written on every pass, including zeros, so they add a fixed handful of series per Instance and never grow with rotation count. Metric label sets are cleaned up when CRs are deleted, so terminated engines do not leave stale series.