Monitoring

Prometheus metrics exposed by the Firebolt Operator and managed components.

This document describes how the Firebolt Operator exposes Prometheus metrics for the components it manages.

Metrics endpoints

Component	Port	Name	Path	What it exposes
Engine pods	9090	metrics	/metrics	firebolt_running_queries, firebolt_suspended_queries, and other Engine gauges. The Firebolt Operator reads the first two for drain checks and auto-stop.
Gateway pods (Envoy)	9090 (default)	metrics	/stats/prometheus	Envoy connection, request, and cluster stats
Firebolt Operator pod	Configurable via metrics.bindAddress	https or http	/metrics	controller-runtime reconciliation, workqueue, REST client, and Go runtime metrics

The gateway metrics port defaults to 9090 and is configurable per FireboltInstance CR via spec.gateway.metricsPort. Metadata pods do not currently expose a Prometheus metrics endpoint.

Firebolt Operator metrics mode

The Firebolt Operator metrics endpoint mode is controlled by two Helm values:

Mode	metrics.secure	metrics.bindAddress	Port name	Scheme
HTTPS (default)	true	:8443	https	https with authn/authz and self-signed TLS
HTTP	false	:8080	http	plain http

The Firebolt Operator PodMonitor template automatically adapts its port reference, scheme, bearer token, and TLS configuration based on metrics.secure.

Scraping with Prometheus

The Firebolt Operator Helm chart ships optional PodMonitor resources (one per component type) that can be enabled via values.yaml:

podMonitor:
  engines:
    enabled: true
  gateway:
    enabled: true
  operator:
    enabled: true
  allNamespaces: false   # set true when the Firebolt Operator watches all namespaces

Each PodMonitor uses label selectors to match the relevant pods:

• Engines: firebolt.io/engine (exists). Matches all engine pods regardless of engine name
• Gateway: firebolt.io/component=gateway
• Firebolt Operator: control-plane=controller-manager + chart selector labels

When allNamespaces is true, namespaceSelector.any: true is added so pods in any namespace are discovered. This does not apply to the Firebolt Operator PodMonitor because the Firebolt Operator always runs in the release namespace.

Per-instance monitoring

The chart-level PodMonitors apply uniform scrape configuration to all instances in scope. If you need per-instance control (different intervals, selective enablement, custom relabelings), disable the chart-level PodMonitors and deploy your own alongside each FireboltInstance or FireboltEngine CR. The label selectors to use are:

• Engine pods: firebolt.io/engine: <engine-name>
• Gateway pods: firebolt.io/instance: <instance-name>, firebolt.io/component: gateway

The chart creates PodMonitors only when you enable them and the Prometheus Operator CRDs are installed. Customize intervals, relabeling, and TLS settings on your own PodMonitors when the chart defaults do not fit your monitoring platform.

The Gateway exposes a read-only stats listener on its metrics port. Its Envoy administration interface remains loopback-only and is not a Prometheus target.

Operator metrics reference

FireboltEngine metrics

Metric	Type	Labels	Updated	Description
firebolt_engine_status_phase	Gauge	namespace, name, instance, phase	Every reconcile	StateSet-style: 1 for the current phase, 0 for all others. Phases: stable, creating, switching, draining, cleaning, stopped.
firebolt_engine_status_condition	Gauge	namespace, name, instance, type	Every reconcile	1 when the condition is True, 0 when False or Unknown. Types: Ready, InstanceReady.
firebolt_engine_spec_replicas	Gauge	namespace, name, instance	Every reconcile	Desired replica count from spec.replicas.
firebolt_engine_active_generation	Gauge	namespace, name, instance	Every reconcile	Generation number currently serving traffic.
firebolt_engine_pods_ready	Gauge	namespace, name, instance	Every reconcile	Number of ready pods in the active generation.
firebolt_engine_pods_total	Gauge	namespace, name, instance	Every reconcile	Total pods in the active generation (includes non-ready).
firebolt_engine_draining_generation	Gauge	namespace, name, instance	Every reconcile	Generation being drained, or -1 if no drain is in progress.
firebolt_engine_last_reconciled_timestamp	Gauge	namespace, name, instance	Successful reconciles	Unix timestamp of the last successful reconcile.
firebolt_engine_drain_check_errors_total	Counter	namespace, name, instance	On drain probe failure	Cumulative count of drain probe failures (pod unreachable, metrics missing).

FireboltInstance metrics

Metric	Type	Labels	Updated	Description
firebolt_instance_status_phase	Gauge	namespace, name, phase	Every reconcile	StateSet-style: 1 for the current phase, 0 for all others. Phases: Provisioning, Ready, Degraded, Failed.
firebolt_instance_status_condition	Gauge	namespace, name, type	Every reconcile	1 when the condition is True, 0 when False or Unknown. Types: Ready, MetadataReady, GatewayReady, AuthReady, EngineTLSReady, GatewayTLSReady.
firebolt_instance_info	Gauge	namespace, name, id, postgres_mode	Every reconcile	Always 1. Carries static metadata: instance ID and postgres mode (internal or external).
firebolt_instance_last_reconciled_timestamp	Gauge	namespace, name	Successful reconciles	Unix timestamp of the last successful reconcile.

JWT signing-key rotation metrics

Present on every FireboltInstance, and flat zero on one that does not use auth.

Metric	Type	Labels	Updated	Description
firebolt_instance_signing_key_generation	Gauge	namespace, name	Every reconcile	Monotonic signing-key generation counter. Increments once per completed mint, so a rotation that ran stays visible afterwards.
firebolt_instance_signing_keys	Gauge	namespace, name, phase	Every reconcile	Keys currently tracked, by phase (Active, ValidationOnly, Removing). Exactly one Active; a non-zero ValidationOnly or Removing means a rotation is in flight.
firebolt_instance_signing_key_rotation_pending_step	Gauge	namespace, name, step	Every reconcile	1 for the rotation step awaiting engine convergence, 0 for the others. All zero when nothing is waiting. Steps: AwaitingPromotion, AwaitingRetireAnchor, AwaitingRemoval.
firebolt_instance_signing_key_rotation_pending_seconds	Gauge	namespace, name	Every reconcile	How long the pending step has been waiting, 0 when none is. Rotation gates park indefinitely by design, so duration, not existence, is what separates a normal engine roll from a stuck fleet.
firebolt_instance_signing_key_rotation_lagging_engines	Gauge	namespace, name	Every reconcile	Engines whose observed auth hash has not converged for the pending step. The true total, not the truncated name list in status.auth.laggingEngines.

Label glossary

Label	Meaning
namespace	Kubernetes namespace of the CR
name	Name of the FireboltEngine or FireboltInstance CR
instance	Name of the parent FireboltInstance (from spec.instanceRef on engines)
phase	Current lifecycle phase
type	Condition type (e.g., Ready, MetadataReady)
step	Signing-key rotation step awaiting engine convergence
id	Stable instance ID (ULID)
postgres_mode	internal (Firebolt Operator-managed) or external (user-provided)

Example PromQL queries

Engine not ready:

firebolt_engine_status_condition{type="Ready"} == 0

Engine stuck in draining phase for more than 10 minutes:

firebolt_engine_status_phase{phase="draining"} == 1
  and firebolt_engine_status_phase{phase="draining"} offset 10m == 1

Scaling in progress (ready pods less than desired):

firebolt_engine_pods_ready < firebolt_engine_spec_replicas

Instance degraded:

firebolt_instance_status_phase{phase="Degraded"} == 1

Stuck controller (no reconcile for 5 minutes):

time() - firebolt_engine_last_reconciled_timestamp > 300

Drain probe failures spiking:

rate(firebolt_engine_drain_check_errors_total[5m]) > 0

Fleet overview (ready engines per instance):

count by (namespace, instance) (firebolt_engine_status_condition{type="Ready"} == 1)

Signing-key rotation stalled for more than 30 minutes:

firebolt_instance_signing_key_rotation_pending_seconds > 1800

Which step, and how many engines it is waiting for:

firebolt_instance_signing_key_rotation_pending_step == 1
firebolt_instance_signing_key_rotation_lagging_engines > 0

A rotation is in flight (more than one key tracked):

sum by (namespace, name) (firebolt_instance_signing_keys) > 1

Alerting on a stalled rotation

Every irreversible rotation step waits for the whole engine fleet to converge, and waits indefinitely — parking is the safe behaviour, so a stuck rotation raises no error and leaves AuthReady True. That makes elapsed time the signal worth alerting on. A normal rotation parks each gate only for as long as one engine roll takes, so a threshold well above your rollout time separates the two:

- alert: FireboltSigningKeyRotationStalled
  expr: firebolt_instance_signing_key_rotation_pending_seconds > 1800
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: >-
      Signing-key rotation on {{ $labels.namespace }}/{{ $labels.name }} has not
      advanced for over 30 minutes.
    description: >-
      An engine has not converged on the current auth configuration. Check
      status.auth.pendingRotationStep and status.auth.laggingEngines on the
      FireboltInstance, then the named engines' own conditions.

The Firebolt Operator also emits a SigningKeyRotationStalled Warning Event when a rotation remains parked past its threshold.

Cardinality

Each FireboltEngine produces approximately 15 time series (6 phases + 2 conditions + 7 scalar gauges). Each FireboltInstance produces approximately 21 series (4 phases + 6 conditions + 1 info + 1 timestamp + 3 signing-key phases + 3 rotation steps + 3 scalar rotation gauges). For a cluster with 10 Instances and 50 Engines, expect roughly 960 time series from the Firebolt Operator.

The phase and step labels on the signing-key metrics are bounded enumerations written on every pass, including zeros, so they add a fixed handful of series per Instance and never grow with rotation count.

Metric label sets are cleaned up when CRs are deleted, so terminated engines do not leave stale series.