Skip to content

Production Monitoring

Observability introduces FraiseQL's three monitoring pieces: error tracking in PostgreSQL, Prometheus metrics and OpenTelemetry tracing, and how to set each one up. This page covers running them in production: managing the errors you track and their notifications, the metrics FraiseQL doesn't record for you, logging, query performance, the database pool, alerts and dashboards.

Need What FraiseQL provides What you add
Error tracking PostgreSQLErrorTracker, the tables of schema.sql, email / Slack / webhook notifications The capture_exception calls, notification rules
Metrics setup_metrics: HTTP metrics and a GET /metrics endpoint Per-operation, pool and cache metrics, and metrics of your own
Tracing setup_tracing: a span per HTTP request, psycopg spans Spans in your resolvers
Logging Standard logging loggers under fraiseql.* Formatter, handlers, request logging
Query performance Query depth / complexity limits, v_query_stats and QueryStatsCollector over pg_stat_statements PostgreSQL's slow-statement log
Health HealthCheck, check_database, check_pool_stats (see Health Checks)
Alerting Error notifications Prometheus alert rules and Alertmanager
Dashboards Grafana dashboards in grafana/

Error Tracking

Setting up the tracker, capturing errors and reading them back are in Observability. In short, start the tracker on the app's pool in a lifespan passed to create_fraiseql_app:

from contextlib import asynccontextmanager

from fraiseql.fastapi import create_fraiseql_app
from fraiseql.fastapi.dependencies import get_db_pool
from fraiseql.monitoring import init_error_tracker


@asynccontextmanager
async def lifespan(app):
    init_error_tracker(get_db_pool(), environment="production", release_version="2.4.1")
    yield


app = create_fraiseql_app(
    database_url="postgresql://localhost/mydb", types=[User], queries=[users], lifespan=lifespan
)

FraiseQL captures nothing on its own: call await get_error_tracker().capture_exception(error, context=...) where you handle errors. A capture that fails (no schema.sql, no partition for the month) is logged and returns ""; it never raises.

Managing Errors

resolve_error marks an error resolved. A new occurrence of a resolved error sets it back to unresolved:

from fraiseql.monitoring import get_error_tracker

tracker = get_error_tracker()
for error in await tracker.get_unresolved_errors(limit=50):
    if error["error_type"] == "PaymentTimeout":
        await tracker.resolve_error(error["error_id"], resolved_by="alice", resolution_notes="Gateway fixed")

The tracker has no methods for the other statuses (ignored, investigating) or for assigning an error: update tb_error_log directly.

-- Assign an error and mark it as being investigated
UPDATE tb_error_log
SET status = 'investigating', assigned_to = 'dev@example.com'
WHERE error_id = '...';

-- Ignore a known error
UPDATE tb_error_log
SET status = 'ignored'
WHERE error_type = 'ExternalApiTimeout';

ignored is a label: an ignored error is still captured, counted, and notified by the rules that match it. To stop its notifications, narrow or disable those rules (see Notifications).

Querying Errors

Errors are rows in tb_error_log (one per fingerprint) and tb_error_occurrence (one per capture, partitioned by month on occurred_at). More queries are in Observability.

-- One error, with its last 10 occurrences
SELECT e.error_type, e.error_message, e.status, e.occurrence_count,
       o.occurred_at, o.request_context, o.user_context, o.stack_trace
FROM tb_error_log e
JOIN tb_error_occurrence o ON o.error_id = e.error_id
WHERE e.error_fingerprint = '...'
  AND o.occurred_at > NOW() - INTERVAL '30 days'
ORDER BY o.occurred_at DESC
LIMIT 10;

-- Users with the most errors this week (from context={"user": {"user_id": ...}})
SELECT o.user_context->>'user_id' AS user_id,
       COUNT(*) AS occurrences,
       array_agg(DISTINCT e.error_type) AS error_types
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
WHERE o.occurred_at > NOW() - INTERVAL '7 days'
  AND o.user_context ? 'user_id'
GROUP BY 1
ORDER BY occurrences DESC
LIMIT 20;

-- Occurrences per hour, last 24 hours
SELECT date_trunc('hour', occurred_at) AS hour, COUNT(*) AS occurrences
FROM tb_error_occurrence
WHERE occurred_at > NOW() - INTERVAL '24 hours'
GROUP BY hour
ORDER BY hour;

-- Totals, unresolved errors, distinct types, mean time to resolve (hours)
SELECT * FROM get_error_stats(24);

Notifications

When the tracker captures an error (enable_notifications=True, the default), it sends a notification for each enabled rule of tb_error_notification_config that matches the error, in a background task. The full description, and a custom channel example, are in Observability. The columns of a rule:

Column Meaning
error_fingerprint, error_type Match this fingerprint / exception class name; NULL matches every error
severity, environment Arrays; the error's severity / environment must be in them; NULL matches every error
min_occurrence_count Send only once the error has occurred this many times (default 1)
rate_limit_minutes At most one notification per error and rule in this window (default 60; 0 for no limit)
channel_type email, slack, webhook, or a channel registered with NotificationManager.register_channel
channel_config JSONB passed to the channel (below)
enabled Rules with false are skipped

The channel_config keys of the built-in channels:

channel_type Keys
email to (list, required), subject (default "Error Alert: {error_type}"), template, and the SMTP settings smtp_host (required), smtp_port (587), smtp_user, smtp_password, use_tls (true), from_address
slack webhook_url (required), channel, username
webhook url (required), method (default POST), headers
-- Errors of the payment code: once they have occurred 10 times, then at most once an hour
INSERT INTO tb_error_notification_config
    (error_type, environment, min_occurrence_count, rate_limit_minutes, channel_type, channel_config)
VALUES
    ('PaymentError', ARRAY['production'], 10, 60, 'webhook',
     '{"url": "https://hooks.example.com/errors", "headers": {"Authorization": "Bearer ..."}}');

-- Stop a rule
UPDATE tb_error_notification_config SET enabled = false WHERE config_id = '...';

There is no rule for "every Nth occurrence": min_occurrence_count sets when the first notification is sent and rate_limit_minutes how often the next ones are.

The webhook channel sends the error as JSON: error_id, error_fingerprint, error_type, error_message, severity, occurrence_count, first_seen, last_seen (ISO 8601), environment, release_version and stack_trace. A 2xx response counts as sent.

Every attempt is logged in tb_error_notification_log, with the reason of a failure:

SELECT n.sent_at, n.channel_type, n.recipient, n.status, n.error_message, e.error_type
FROM tb_error_notification_log n
JOIN tb_error_log e ON e.error_id = n.error_id
WHERE n.sent_at > NOW() - INTERVAL '24 hours'
ORDER BY n.sent_at DESC;

tb_error_notification_log is partitioned by month, like tb_error_occurrence: run SELECT ensure_monitoring_partitions(2); every month (see Retention and Partitions).

Errors and Traces

With tracing on, pass the current span's ids to capture_exception; the error's trace_id then finds the request in your tracing backend (Jaeger, Tempo, ...):

from opentelemetry import trace

import fraiseql
from fraiseql.monitoring import get_error_tracker


@fraiseql.query
async def invoice(info, id: int) -> Invoice | None:
    try:
        return await info.context["db"].find_one("v_invoice", mandatory_filters={"id": id})
    except Exception as error:
        span = trace.get_current_span().get_span_context()
        await get_error_tracker().capture_exception(
            error,
            context={"request": {"operation": "invoice", "invoice_id": id}},
            trace_id=format(span.trace_id, "032x"),
            span_id=format(span.span_id, "016x"),
        )
        raise
SELECT error_type, error_message, last_seen, trace_id
FROM tb_error_log
WHERE trace_id IS NOT NULL
ORDER BY last_seen DESC
LIMIT 20;

schema.sql also creates otel_traces and otel_metrics, but nothing in FraiseQL writes to them: spans go to the exporter of setup_tracing.

Coming from Sentry

Sentry FraiseQL
sentry_sdk.init(dsn=..., environment=..., release=...) init_error_tracker(pool, environment=..., release_version=...), after loading schema.sql
sentry_sdk.capture_exception(error) await get_error_tracker().capture_exception(error)
sentry_sdk.capture_message(text) await get_error_tracker().capture_message(text, level="warning")
set_user, set_context, set_tag capture_exception(error, context={"user": {...}, "request": {...}, "application": {...}}, tags=[...])
Issue grouping Fingerprint of the exception type and the frame where it was raised, or context={"fingerprint": "..."}
Alert rules Rows of tb_error_notification_config
traces_sample_rate TracingConfig(sample_rate=...) with setup_tracing

What isn't provided: automatic capture of every unhandled exception, a web UI (use SQL or the error_monitoring.json Grafana dashboard), release health, and front-end error tracking.

Caching

PostgresCache (fraiseql.caching) is a key-value cache in an UNLOGGED table (fraiseql_cache by default, created on first use). UNLOGGED tables skip the write-ahead log, so writes are cheaper than in a regular table, but PostgreSQL empties them after a crash and doesn't replicate them: treat the content as disposable. All app instances on the same database share it. The full guide, including CachedRepository, is Caching.

from fraiseql.caching import PostgresCache

cache = PostgresCache(pool)

await cache.set("user:123", {"name": "Alice"}, ttl=3600)  # ttl in seconds, required
value = await cache.get("user:123")                       # None when missing or expired
await cache.delete("user:123")
await cache.delete_pattern("user:*")                      # * (or %) is a wildcard

removed = await cache.cleanup_expired()  # expired rows stay until you call this
stats = await cache.get_stats()
# {'total_entries': ..., 'expired_entries': ..., 'active_entries': ..., 'table_size_bytes': ...}

Expired entries are never returned, but they stay in the table until cleanup_expired() runs: call it periodically, from a background task or a scheduler. There is no batch get_many / set_many.

Coming from Redis: redis.set(key, value, ex=ttl) is cache.set(key, value, ttl=ttl), get and delete are the same, and deleting the keys of redis.keys(pattern) is cache.delete_pattern(pattern). Values must be JSON-serializable.

Metrics

setup_metrics(app) records the HTTP metrics of every request and serves GET /metrics (see Observability). The metrics FraiseQL exports, and which of them it records itself, are listed in the runbooks' metrics reference. The others get values only from your code, as below.

Per-Operation Metrics

All GraphQL operations share the /graphql endpoint. For one series per resolver, decorate it with @with_metrics("query") under the FraiseQL decorator:

import fraiseql
from fraiseql.monitoring import with_metrics


@fraiseql.query
@with_metrics("query")
async def users(info) -> list[User]:
    return await info.context["db"].find("v_user")

Each call increments fraiseql_graphql_queries_total{operation_type="query", operation_name="users"}, observes fraiseql_graphql_query_duration_seconds with the same labels, and increments fraiseql_graphql_queries_success_total or fraiseql_graphql_queries_errors_total (label operation_type only). An exception raised by the resolver is also counted in fraiseql_errors_total, with operation set to the function name. @with_metrics("mutation") on a function mutation records the same metrics with operation_type="mutation".

# Calls per second, by resolver
sum by (operation_name) (rate(fraiseql_graphql_queries_total[5m]))

# p95 duration, by resolver
histogram_quantile(0.95, sum by (le, operation_name) (rate(fraiseql_graphql_query_duration_seconds_bucket[5m])))

Database Pool Metrics

The fraiseql_db_connections_active, _idle and _total gauges read 0 until your code sets them. Sample the app's pool from a task started in the app's lifespan:

import asyncio
from contextlib import asynccontextmanager

from fraiseql.fastapi import create_fraiseql_app
from fraiseql.fastapi.dependencies import get_db_pool
from fraiseql.monitoring import setup_metrics


async def report_pool(pool, metrics, interval: float = 15) -> None:
    while True:
        stats = pool.get_stats()
        size, idle = stats.get("pool_size", 0), stats.get("pool_available", 0)
        metrics.update_db_connections(active=size - idle, idle=idle, total=size)
        await asyncio.sleep(interval)


@asynccontextmanager
async def lifespan(app):
    task = asyncio.create_task(report_pool(get_db_pool(), metrics))
    yield
    task.cancel()


app = create_fraiseql_app(
    database_url="postgresql://localhost/mydb", types=[User], queries=[users], lifespan=lifespan
)
metrics = setup_metrics(app)

pool_size is the number of connections the pool holds, pool_available the idle ones; the pool's limit is pool.max_size.

Metrics of Your Own

/metrics serves the registry of the FraiseQLMetrics instance that setup_metrics returns (metrics.registry), not prometheus_client's default one. Register your own metrics on it to have them served:

from prometheus_client import Counter, ProcessCollector

metrics = setup_metrics(app)

orders_created = Counter(
    "myapp_orders_created_total", "Orders created", ["channel"], registry=metrics.registry
)
ProcessCollector(registry=metrics.registry)  # process_cpu_seconds_total, process_resident_memory_bytes, ...

orders_created.labels(channel="web").inc()

FraiseQLMetrics also has record_mutation(mutation_name, duration_ms, success, result_type=None, error_type=None), record_db_query(query_type, table_name, duration_ms) and record_cache_hit(cache_type) / record_cache_miss(cache_type) for the mutation, database and cache metrics.

Logging

FraiseQL logs with the standard logging module, on loggers named after its modules (fraiseql.fastapi.routers, fraiseql.monitoring.notifications, ...), and adds no handler to them: your application's logging configuration decides the format and the destination. To set FraiseQL's level apart from your own:

import logging

logging.getLogger("fraiseql").setLevel(logging.WARNING)

The exception is the security log (fraiseql.security: authentication, rate limiting, rejected queries). The first security event sets that logger to INFO and gives it two handlers of its own, stderr and the file security_events.log in the working directory (FRAISEQL_SECURITY_LOG_PATH to change it); its records also propagate to your handlers. Each record's message is the event as JSON. To send security events only through your own handlers, set the security logger yourself before the first event:

from fraiseql.audit import SecurityLogger, set_security_logger

set_security_logger(SecurityLogger(log_to_file=False, log_to_stdout=False))

Structured Logging

A formatter that writes one JSON object per line, with the extra fields you pass:

import json
import logging
from datetime import UTC, datetime

STANDARD_ATTRS = set(vars(logging.LogRecord("", 0, "", 0, "", None, None))) | {"message", "asctime"}


class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        entry = {
            "timestamp": datetime.fromtimestamp(record.created, UTC).isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        # Fields passed with logger.info(..., extra={...})
        entry.update({k: v for k, v in vars(record).items() if k not in STANDARD_ATTRS})
        if record.exc_info:
            entry["exception"] = self.formatException(record.exc_info)
        return json.dumps(entry, default=str)


handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

logger = logging.getLogger("myapp")
logger.info("Order created", extra={"order_id": "order-456", "duration_ms": 125.5})
# {"timestamp": "...", "level": "INFO", "logger": "myapp", "message": "Order created", "order_id": "order-456", "duration_ms": 125.5}

Request Logging

FraiseQL doesn't log requests or assign request ids. A middleware on the app does both:

import logging
import time
import uuid

from starlette.middleware.base import BaseHTTPMiddleware

logger = logging.getLogger("myapp.requests")


class RequestLoggingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        request_id = request.headers.get("X-Request-ID") or str(uuid.uuid4())
        request.state.request_id = request_id
        start = time.perf_counter()
        try:
            response = await call_next(request)
        except Exception:
            logger.exception("Request failed", extra={"request_id": request_id, "path": request.url.path})
            raise
        logger.info(
            "Request completed",
            extra={
                "request_id": request_id,
                "method": request.method,
                "path": request.url.path,
                "status_code": response.status_code,
                "duration_ms": round((time.perf_counter() - start) * 1000, 1),
            },
        )
        response.headers["X-Request-ID"] = request_id
        return response


app.add_middleware(RequestLoggingMiddleware)

A GraphQL error comes back with status 200, so this logs it as a success; errors inside the response are for the error tracker. For log aggregation, see Loki Integration.

Query Performance

Request Latency

fraiseql_http_request_duration_seconds times every request, by method and endpoint:

# p95 latency of /graphql
histogram_quantile(0.95, sum by (le) (rate(fraiseql_http_request_duration_seconds_bucket{endpoint="/graphql"}[5m])))

For latency by operation, use @with_metrics (see Per-Operation Metrics) or tracing.

Query Depth and Complexity

FraiseQL can reject a query that is too deep or too expensive before running it. The limits are off by default; set them on FraiseQLConfig:

from fraiseql.fastapi import FraiseQLConfig

config = FraiseQLConfig(
    database_url="postgresql://localhost/mydb",
    complexity_enabled=True,
    complexity_max_score=1000,
    complexity_max_depth=10,
)

A rejected query gets a GraphQL error with code QUERY_TOO_COMPLEX or QUERY_TOO_DEEP, and a query.complexity_exceeded / query.depth_exceeded entry in the security log. There is no metric for them. To see the score of a query before you choose a limit, measure it with the same code:

from fraiseql import build_fraiseql_schema
from fraiseql.gql.query_limits import QueryLimits

schema = build_fraiseql_schema(query_types=[users])
cost = QueryLimits().measure(schema, "{ users(limit: 50) { id name } }")
print(cost.depth, cost.complexity)

A scalar field costs 1, an object field 1 plus its selections, and a list field 1 plus its selections times the list size (its limit or first argument, else complexity_default_list_size, 10).

Slow SQL

FraiseQL has no SQL echo setting (database_echo has no effect). Log statements in PostgreSQL:

-- Development: log every statement
ALTER DATABASE myapp SET log_statement = 'all';

-- Production: log slow statements only (> 1s)
ALTER DATABASE myapp SET log_min_duration_statement = 1000;

With pg_stat_statements installed (shared_preload_libraries = 'pg_stat_statements', then CREATE EXTENSION pg_stat_statements) before schema.sql is loaded, the schema also creates the view v_query_stats, and QueryStatsCollector reads it:

from fraiseql.monitoring import init_query_stats

collector = init_query_stats(pool)
if await collector.is_available():
    for s in await collector.get_stats(top_n=10, order_by="mean_exec_time"):
        print(f"{s.mean_exec_time_ms:8.1f} ms  {s.calls:6} calls  {s.query_preview[:80]}")

order_by is one of total_exec_time, mean_exec_time, calls and cache_hit_ratio. Without the extension, get_stats returns an empty list.

Database Monitoring

The app's pool is a psycopg AsyncConnectionPool, returned by fraiseql.fastapi.dependencies.get_db_pool() once the app has started. pool.get_stats() reports its counters (pool_size, pool_available, requests_waiting, requests_num, ...); feed them to Prometheus as in Database Pool Metrics, or expose them on a health endpoint with check_pool_stats (see Health Checks).

PostgreSQL's own view of the connections, whatever the app reports:

SELECT state, COUNT(*) AS connections, MAX(NOW() - state_change) AS longest_in_state
FROM pg_stat_activity
WHERE datname = current_database()
  AND backend_type = 'client backend'
GROUP BY state;

Set application_name in the connection string (postgresql://.../mydb?application_name=orders-api) to tell the app's connections apart. The database_pool.json Grafana dashboard shows these views.

Alerting

FraiseQL has no alerting of its own besides error notifications. Alert on the metrics with Prometheus rules, and route them (Slack, email, PagerDuty, ...) with Alertmanager. These rules use only metrics setup_metrics records:

# prometheus-alerts.yml
groups:
  - name: fraiseql
    rules:
      - alert: FraiseQLHigh5xxRate
        expr: |
          sum(rate(fraiseql_http_requests_total{status=~"5.."}[5m]))
            / sum(rate(fraiseql_http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "More than 5% of requests fail with a 5xx status"
          description: "5xx ratio is {{ $value | humanizePercentage }}"

      - alert: FraiseQLHighLatency
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(fraiseql_http_request_duration_seconds_bucket{endpoint="/graphql"}[5m]))
          ) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p99 latency of /graphql above 1s"
          description: "p99 latency is {{ $value | humanizeDuration }}"

      - alert: FraiseQLUnhandledExceptions
        expr: sum by (error_type) (increase(fraiseql_errors_total[10m])) > 0
        labels:
          severity: warning
        annotations:
          summary: "Unhandled {{ $labels.error_type }} exceptions"

With the pool gauges fed (see Database Pool Metrics), add a rule on them; the limit isn't exported, so write it in the rule:

      - alert: FraiseQLPoolSaturated
        expr: fraiseql_db_connections_active / 20 > 0.9  # 20 = the pool's max_size
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Database pool more than 90% used"

A GraphQL error (a failed resolver, a rejected query) comes back with status 200 and counts in none of these metrics. Alert on those with error notifications, or with @with_metrics and fraiseql_graphql_queries_errors_total.

Dashboards

The grafana/ directory of the repository has four dashboards, and a script to import them (see grafana/README.md):

Dashboard Datasource Shows
error_monitoring.json PostgreSQL tb_error_log, tb_error_occurrence, tb_error_notification_log
performance_metrics.json Prometheus The HTTP metrics of setup_metrics
database_pool.json PostgreSQL pg_stat_activity, pg_stat_database, pg_stat_user_tables, v_query_stats
apq_effectiveness.json PostgreSQL The tables of the PostgreSQL APQ backend

Next Steps