> ## Documentation Index > Fetch the complete documentation index at: https://firebolt-aggregate-helm-docs-pr-79.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring > Prometheus metrics exposed by the Firebolt Operator and managed components. This document describes how the Firebolt Operator exposes Prometheus metrics for the components it manages. ## Metrics endpoints | Component | Port | Name | Path | What it exposes | | --------------------- | -------------------------------------- | ----------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Engine pods | 9090 | `metrics` | `/metrics` | `firebolt_running_queries`, `firebolt_suspended_queries`, and other Engine gauges. The Firebolt Operator reads the first two for drain checks and auto-stop. | | Gateway pods (Envoy) | 9090 (default) | `metrics` | `/stats/prometheus` | Envoy connection, request, and cluster stats | | Firebolt Operator pod | Configurable via `metrics.bindAddress` | `https` or `http` | `/metrics` | controller-runtime reconciliation, workqueue, REST client, and Go runtime metrics | The gateway metrics port defaults to 9090 and is configurable per FireboltInstance CR via `spec.gateway.metricsPort`. Metadata pods do not currently expose a Prometheus metrics endpoint. ### Firebolt Operator metrics mode The Firebolt Operator metrics endpoint mode is controlled by two Helm values: | Mode | `metrics.secure` | `metrics.bindAddress` | Port name | Scheme | | --------------- | ---------------- | --------------------- | --------- | -------------------------------------------- | | HTTPS (default) | `true` | `:8443` | `https` | `https` with authn/authz and self-signed TLS | | HTTP | `false` | `:8080` | `http` | plain `http` | The Firebolt Operator PodMonitor template automatically adapts its port reference, scheme, bearer token, and TLS configuration based on `metrics.secure`. ## Scraping with Prometheus The Firebolt Operator Helm chart ships optional `PodMonitor` resources (one per component type) that can be enabled via `values.yaml`: ```yaml theme={"theme":{"light":"css-variables","dark":"css-variables"}} podMonitor: engines: enabled: true gateway: enabled: true operator: enabled: true allNamespaces: false # set true when the Firebolt Operator watches all namespaces ``` Each PodMonitor uses label selectors to match the relevant pods: * **Engines**: `firebolt.io/engine` (exists). Matches all engine pods regardless of engine name * **Gateway**: `firebolt.io/component=gateway` * **Firebolt Operator**: `control-plane=controller-manager` + chart selector labels When `allNamespaces` is true, `namespaceSelector.any: true` is added so pods in any namespace are discovered. This does not apply to the Firebolt Operator PodMonitor because the Firebolt Operator always runs in the release namespace. ### Per-instance monitoring The chart-level PodMonitors apply uniform scrape configuration to all instances in scope. If you need per-instance control (different intervals, selective enablement, custom relabelings), disable the chart-level PodMonitors and deploy your own alongside each FireboltInstance or FireboltEngine CR. The label selectors to use are: * Engine pods: `firebolt.io/engine: ` * Gateway pods: `firebolt.io/instance: `, `firebolt.io/component: gateway` The chart creates PodMonitors only when you enable them and the Prometheus Operator CRDs are installed. Customize intervals, relabeling, and TLS settings on your own PodMonitors when the chart defaults do not fit your monitoring platform. The Gateway exposes a read-only stats listener on its metrics port. Its Envoy administration interface remains loopback-only and is not a Prometheus target. ## Operator metrics reference ### FireboltEngine metrics | Metric | Type | Labels | Updated | Description | | ------------------------------------------- | ------- | ---------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `firebolt_engine_status_phase` | Gauge | `namespace`, `name`, `instance`, `phase` | Every reconcile | StateSet-style: 1 for the current phase, 0 for all others. Phases: `stable`, `creating`, `switching`, `draining`, `cleaning`, `stopped`. | | `firebolt_engine_status_condition` | Gauge | `namespace`, `name`, `instance`, `type` | Every reconcile | 1 when the condition is True, 0 when False or Unknown. Types: `Ready`, `InstanceReady`. | | `firebolt_engine_spec_replicas` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Desired replica count from `spec.replicas`. | | `firebolt_engine_active_generation` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Generation number currently serving traffic. | | `firebolt_engine_pods_ready` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Number of ready pods in the active generation. | | `firebolt_engine_pods_total` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Total pods in the active generation (includes non-ready). | | `firebolt_engine_draining_generation` | Gauge | `namespace`, `name`, `instance` | Every reconcile | Generation being drained, or -1 if no drain is in progress. | | `firebolt_engine_last_reconciled_timestamp` | Gauge | `namespace`, `name`, `instance` | Successful reconciles | Unix timestamp of the last successful reconcile. | | `firebolt_engine_drain_check_errors_total` | Counter | `namespace`, `name`, `instance` | On drain probe failure | Cumulative count of drain probe failures (pod unreachable, metrics missing). | ### FireboltInstance metrics | Metric | Type | Labels | Updated | Description | | --------------------------------------------- | ----- | ------------------------------------------ | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `firebolt_instance_status_phase` | Gauge | `namespace`, `name`, `phase` | Every reconcile | StateSet-style: 1 for the current phase, 0 for all others. Phases: `Provisioning`, `Ready`, `Degraded`, `Failed`. | | `firebolt_instance_status_condition` | Gauge | `namespace`, `name`, `type` | Every reconcile | 1 when the condition is True, 0 when False or Unknown. Types: `Ready`, `MetadataReady`, `GatewayReady`, `AuthReady`, `EngineTLSReady`, `GatewayTLSReady`. | | `firebolt_instance_info` | Gauge | `namespace`, `name`, `id`, `postgres_mode` | Every reconcile | Always 1. Carries static metadata: instance ID and postgres mode (`internal` or `external`). | | `firebolt_instance_last_reconciled_timestamp` | Gauge | `namespace`, `name` | Successful reconciles | Unix timestamp of the last successful reconcile. | ### JWT signing-key rotation metrics Present on every FireboltInstance, and flat zero on one that does not use auth. | Metric | Type | Labels | Updated | Description | | -------------------------------------------------------- | ----- | ---------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `firebolt_instance_signing_key_generation` | Gauge | `namespace`, `name` | Every reconcile | Monotonic signing-key generation counter. Increments once per completed mint, so a rotation that ran stays visible afterwards. | | `firebolt_instance_signing_keys` | Gauge | `namespace`, `name`, `phase` | Every reconcile | Keys currently tracked, by phase (`Active`, `ValidationOnly`, `Removing`). Exactly one `Active`; a non-zero `ValidationOnly` or `Removing` means a rotation is in flight. | | `firebolt_instance_signing_key_rotation_pending_step` | Gauge | `namespace`, `name`, `step` | Every reconcile | 1 for the rotation step awaiting engine convergence, 0 for the others. All zero when nothing is waiting. Steps: `AwaitingPromotion`, `AwaitingRetireAnchor`, `AwaitingRemoval`. | | `firebolt_instance_signing_key_rotation_pending_seconds` | Gauge | `namespace`, `name` | Every reconcile | How long the pending step has been waiting, 0 when none is. Rotation gates park indefinitely by design, so **duration**, not existence, is what separates a normal engine roll from a stuck fleet. | | `firebolt_instance_signing_key_rotation_lagging_engines` | Gauge | `namespace`, `name` | Every reconcile | Engines whose observed auth hash has not converged for the pending step. The true total, not the truncated name list in `status.auth.laggingEngines`. | ### Label glossary | Label | Meaning | | --------------- | ------------------------------------------------------------------------ | | `namespace` | Kubernetes namespace of the CR | | `name` | Name of the FireboltEngine or FireboltInstance CR | | `instance` | Name of the parent FireboltInstance (from `spec.instanceRef` on engines) | | `phase` | Current lifecycle phase | | `type` | Condition type (e.g., `Ready`, `MetadataReady`) | | `step` | Signing-key rotation step awaiting engine convergence | | `id` | Stable instance ID (ULID) | | `postgres_mode` | `internal` (Firebolt Operator-managed) or `external` (user-provided) | ### Example PromQL queries Engine not ready: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_engine_status_condition{type="Ready"} == 0 ``` Engine stuck in draining phase for more than 10 minutes: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_engine_status_phase{phase="draining"} == 1 and firebolt_engine_status_phase{phase="draining"} offset 10m == 1 ``` Scaling in progress (ready pods less than desired): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_engine_pods_ready < firebolt_engine_spec_replicas ``` Instance degraded: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_instance_status_phase{phase="Degraded"} == 1 ``` Stuck controller (no reconcile for 5 minutes): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} time() - firebolt_engine_last_reconciled_timestamp > 300 ``` Drain probe failures spiking: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} rate(firebolt_engine_drain_check_errors_total[5m]) > 0 ``` Fleet overview (ready engines per instance): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} count by (namespace, instance) (firebolt_engine_status_condition{type="Ready"} == 1) ``` Signing-key rotation stalled for more than 30 minutes: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_instance_signing_key_rotation_pending_seconds > 1800 ``` Which step, and how many engines it is waiting for: ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} firebolt_instance_signing_key_rotation_pending_step == 1 firebolt_instance_signing_key_rotation_lagging_engines > 0 ``` A rotation is in flight (more than one key tracked): ```promql theme={"theme":{"light":"css-variables","dark":"css-variables"}} sum by (namespace, name) (firebolt_instance_signing_keys) > 1 ``` ### Alerting on a stalled rotation Every irreversible rotation step waits for the whole engine fleet to converge, and waits indefinitely — parking is the safe behaviour, so a stuck rotation raises no error and leaves `AuthReady` `True`. That makes elapsed time the signal worth alerting on. A normal rotation parks each gate only for as long as one engine roll takes, so a threshold well above your rollout time separates the two: ```yaml theme={"theme":{"light":"css-variables","dark":"css-variables"}} - alert: FireboltSigningKeyRotationStalled expr: firebolt_instance_signing_key_rotation_pending_seconds > 1800 for: 10m labels: severity: warning annotations: summary: >- Signing-key rotation on {{ $labels.namespace }}/{{ $labels.name }} has not advanced for over 30 minutes. description: >- An engine has not converged on the current auth configuration. Check status.auth.pendingRotationStep and status.auth.laggingEngines on the FireboltInstance, then the named engines' own conditions. ``` The Firebolt Operator also emits a `SigningKeyRotationStalled` Warning Event when a rotation remains parked past its threshold. ### Cardinality Each FireboltEngine produces approximately 15 time series (6 phases + 2 conditions + 7 scalar gauges). Each FireboltInstance produces approximately 21 series (4 phases + 6 conditions + 1 info + 1 timestamp + 3 signing-key phases + 3 rotation steps + 3 scalar rotation gauges). For a cluster with 10 Instances and 50 Engines, expect roughly 960 time series from the Firebolt Operator. The `phase` and `step` labels on the signing-key metrics are bounded enumerations written on every pass, including zeros, so they add a fixed handful of series per Instance and never grow with rotation count. Metric label sets are cleaned up when CRs are deleted, so terminated engines do not leave stale series.