Skip to content

Observability

FraiseQL has three observability pieces. None is on by default; each is set up in your application code:

Piece API Where the data goes
Error tracking fraiseql.monitoring.init_error_tracker PostgreSQL tables tb_error_log, tb_error_occurrence
Metrics fraiseql.monitoring.setup_metrics A Prometheus /metrics endpoint
Tracing fraiseql.tracing.setup_tracing An OpenTelemetry exporter (OTLP, Jaeger or Zipkin)

Errors live in your database, so you can query them with SQL and join them with your own tables. Metrics and traces go to the tools built for them (Prometheus, an OpenTelemetry collector).

Error Tracking

PostgreSQLErrorTracker stores exceptions in PostgreSQL, grouped by fingerprint (like Sentry issues), with every occurrence kept in a partitioned table.

Schema

The tables, views and functions are in schema.sql, in the installed fraiseql.monitoring package. FraiseQL doesn't create them: run the file once per database, as part of your migrations.

SCHEMA=$(python -c "import fraiseql.monitoring, pathlib; print(pathlib.Path(fraiseql.monitoring.__file__).parent / 'schema.sql')")
psql "$DATABASE_URL" -f "$SCHEMA"

It creates:

Object What it holds
tb_error_log One row per fingerprint: error_type, error_message, stack_trace, request_context / user_context / application_context (JSONB), severity, tags, environment, release_version, first_seen, last_seen, occurrence_count, status (unresolved, resolved, ignored, investigating), assigned_to, resolved_at, resolved_by, trace_id, span_id
tb_error_occurrence One row per capture (error_id, occurred_at, contexts, stack_trace, breadcrumbs, trace_id), partitioned by month on occurred_at
tb_error_notification_config, tb_error_notification_log Notification rules and their delivery log (see Notifications)
v_active_errors, v_top_errors, v_error_trends Unresolved errors; the most frequent ones; errors per hour
get_error_stats(p_hours), resolve_error(p_error_id, p_resolved_by, p_resolution_notes) SQL counterparts of the tracker methods
ensure_monitoring_partitions(months_ahead), create_monthly_partition(table, date), create_error_occurrence_partition(date), ensure_error_occurrence_partitions(months_ahead), drop_old_error_occurrence_partitions(retention_months), get_partition_stats() Partition management (see Retention)
otel_traces, otel_metrics Tables for spans and metrics, partitioned by month. Nothing in FraiseQL writes to them: traces and metrics go to the exporters below

The file creates the partitions of the four partitioned tables (tb_error_occurrence, tb_error_notification_log, otel_traces, otel_metrics) for the current month and the next two. A capture fails (and is logged, not raised) when its month has no partition, so create them ahead of time (see Retention).

Setup

from psycopg_pool import AsyncConnectionPool

from fraiseql.monitoring import init_error_tracker

pool = AsyncConnectionPool("postgresql://localhost/mydb", open=False)


async def startup() -> None:
    await pool.open()
    init_error_tracker(
        pool,
        environment="production",
        release_version="2.4.1",
        enable_notifications=False,  # see Notifications below
    )

init_error_tracker sets the process-wide tracker that get_error_tracker() returns, and returns it.

Capturing Errors

FraiseQL captures nothing on its own: call capture_exception where you handle errors. A resolver's exception becomes a GraphQL error inside the response and never reaches an HTTP middleware, so capture it in the resolver:

import fraiseql
from fraiseql.monitoring import get_error_tracker


@fraiseql.query
async def invoice(info, id: int) -> Invoice | None:
    try:
        return await info.context["db"].find_one("v_invoice", mandatory_filters={"id": id})
    except Exception as error:
        tracker = get_error_tracker()
        if tracker is not None:
            await tracker.capture_exception(
                error,
                context={
                    "request": {"operation": "invoice", "invoice_id": id},
                    "user": {"user_id": str(info.context.get("user_id"))},
                },
                severity="error",
                tags=["billing"],
            )
        raise

capture_exception(error, context=None, severity="error", tags=None, trace_id=None, span_id=None) returns the error's id. The context keys it reads are request, user and application (stored in the matching JSONB columns), breadcrumbs (stored with the occurrence) and fingerprint (a string that replaces the default grouping). capture_message(message, level="info", context=None, tags=None) records a message the same way.

Errors are grouped by a fingerprint of the exception type and the file, line and function where it was raised (the last frame outside FraiseQL and site-packages). A new occurrence of a known fingerprint increments occurrence_count, updates last_seen, and re-opens a resolved error.

Reading and Resolving Errors

tracker = get_error_tracker()

errors = await tracker.get_unresolved_errors(limit=20, severity="error")
details = await tracker.get_error(errors[0]["error_id"])
stats = await tracker.get_error_stats(hours=24)
# {'total_errors': ..., 'unresolved_errors': ..., 'unique_error_types': ..., 'avg_resolution_time_hours': ...}

await tracker.resolve_error(errors[0]["error_id"], resolved_by="alice", resolution_notes="Fixed in 2.4.2")

Or in SQL:

-- Most frequent unresolved errors
SELECT error_type, error_message, occurrence_count, last_seen
FROM v_top_errors
LIMIT 10;

-- Errors of the last 24 hours, with the users they affected
SELECT
    e.error_type,
    e.error_message,
    COUNT(*) AS occurrences,
    COUNT(DISTINCT o.user_context->>'user_id') AS affected_users,
    MAX(o.occurred_at) AS last_seen
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
WHERE o.occurred_at > NOW() - INTERVAL '24 hours'
GROUP BY e.error_id, e.error_type, e.error_message
ORDER BY occurrences DESC
LIMIT 20;

-- Occurrences per hour over the last week (filter on occurred_at to prune partitions)
SELECT date_trunc('hour', o.occurred_at) AS hour, e.error_type, COUNT(*) AS count
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
WHERE o.occurred_at > NOW() - INTERVAL '7 days'
GROUP BY hour, e.error_type
ORDER BY hour DESC, count DESC;

-- Errors by release
SELECT release_version, COUNT(*) AS errors, SUM(occurrence_count) AS occurrences
FROM tb_error_log
GROUP BY release_version
ORDER BY occurrences DESC;

Because the errors are rows in your database, you can join them with your own tables, for example on an id you put in the context:

SELECT e.error_type, e.error_message, o.occurred_at, i.amount
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
JOIN tb_invoice i ON i.id = (o.request_context->>'invoice_id')::int
WHERE o.occurred_at > NOW() - INTERVAL '24 hours';

Notifications

The tracker sends a notification when it captures an error (enable_notifications=True, the default), in a background task: its NotificationManager reads the enabled rules of tb_error_notification_config that match the error (error_fingerprint, error_type, severity and environment filters, NULL matching everything, and min_occurrence_count), skips a rule that already sent for this error within its rate_limit_minutes, sends through the rule's channel_type (email, slack, webhook, or one added with register_channel), and logs the attempt (sent or failed, with the reason) in tb_error_notification_log.

INSERT INTO tb_error_notification_config
    (error_type, severity, environment, channel_type, channel_config, rate_limit_minutes)
VALUES
    (NULL, ARRAY['error', 'critical'], ARRAY['production'], 'webhook',
     '{"url": "https://hooks.example.com/errors", "headers": {"Authorization": "Bearer ..."}}', 30),
    ('PaymentError', NULL, NULL, 'slack',
     '{"webhook_url": "https://hooks.slack.com/services/...", "channel": "#payments"}', 60),
    (NULL, ARRAY['critical'], NULL, 'email',
     '{"smtp_host": "smtp.example.com", "smtp_user": "alerts", "smtp_password": "...",
       "from_address": "alerts@example.com", "to": ["oncall@example.com"]}', 15);

channel_config is passed to the channel's send; a channel class also gets the keys its constructor takes (EmailChannel its smtp_host, smtp_port, smtp_user, smtp_password, use_tls and from_address). For a channel of your own, register it on a manager and give that manager to the tracker:

from fraiseql.monitoring.notifications import NotificationManager
from fraiseql.monitoring.postgres_error_tracker import init_error_tracker


class PagerChannel:
    async def send(self, error: dict, config: dict) -> tuple[bool, str | None]:
        await page(config["service"], f"{error['error_type']}: {error['error_message']}")
        return True, None  # (sent?, error message)

    def format_message(self, error: dict, template: str | None = None) -> str:
        return error["error_message"]


manager = NotificationManager(pool)
manager.register_channel("pager", PagerChannel)  # a class, or an instance
tracker = init_error_tracker(pool, environment="production", notification_manager=manager)

tb_error_notification_log, like tb_error_occurrence, otel_traces and otel_metrics, is partitioned by month: schema.sql creates the partitions of the current month and the next two. Run SELECT ensure_monitoring_partitions(2); every month (pg_cron or any scheduler), or writes fail once the months run out.

Metrics

setup_metrics(app) adds a middleware that counts and times every HTTP request, and a GET /metrics endpoint for Prometheus (it needs prometheus_client):

from fraiseql.fastapi import create_fraiseql_app
from fraiseql.monitoring import MetricsConfig, setup_metrics

app = create_fraiseql_app(database_url="postgresql://localhost/mydb", types=[User], queries=[users])
metrics = setup_metrics(app, MetricsConfig(namespace="fraiseql"))

The metrics recorded on their own are fraiseql_http_requests_total, fraiseql_http_request_duration_seconds, fraiseql_response_time_seconds and fraiseql_errors_total (exceptions that escape the app). The GraphQL, database pool and cache metrics only get values your code records (@with_metrics, metrics.update_db_connections(...), ...). The full list, and the queries to use them, are in the runbooks' metrics reference.

Distributed Tracing

With the tracing extra (pip install "fraiseql[tracing]": the OpenTelemetry SDK, the OTLP exporter and the psycopg instrumentation), fraiseql.tracing.setup_tracing(app, TracingConfig(...)) opens a span per HTTP request (POST /graphql, kind SERVER, with the http.* attributes and the status code; exclude_paths defaults to /health, /ready, /metrics, /docs, /openapi.json), continues the trace of an incoming traceparent header, instruments psycopg, and exports the spans:

from fraiseql.fastapi import create_fraiseql_app
from fraiseql.tracing import TracingConfig, setup_tracing

app = create_fraiseql_app(database_url="postgresql://localhost/mydb", types=[User], queries=[users])
setup_tracing(
    app,
    TracingConfig(
        service_name="orders-api",
        deployment_environment="production",
        sample_rate=0.1,                                 # keep 10% of traces
        export_format="otlp",                            # or "console", "zipkin"
        export_endpoint="http://otel-collector:4317",    # OTLP over gRPC
    ),
)

export_format="console" prints the spans (no endpoint needed); "zipkin" needs opentelemetry-exporter-zipkin-json. "jaeger" raises an ImportError: its exporter is deprecated upstream and doesn't install with the current SDK; Jaeger accepts OTLP, so use "otlp" with its port 4317. Without export_endpoint (OTLP, Zipkin), spans are recorded but not exported. If a TracerProvider is already set when setup_tracing runs, fraiseql uses it as it is (its exporters, not the config's).

Or set up OpenTelemetry yourself; this traces every SQL statement FraiseQL runs, and spans you open in resolvers:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.psycopg import PsycopgInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

import fraiseql

provider = TracerProvider(
    resource=Resource.create({"service.name": "orders-api", "deployment.environment": "production"}),
    sampler=TraceIdRatioBased(0.1),  # keep 10% of traces
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)))
trace.set_tracer_provider(provider)
PsycopgInstrumentor().instrument()  # before the app opens its pool

tracer = trace.get_tracer("orders-api")


@fraiseql.query
async def users(info) -> list[User]:
    with tracer.start_as_current_span("users"):
        return await info.context["db"].find("v_user")

For a span per HTTP request, add opentelemetry-instrumentation-fastapi and call FastAPIInstrumentor.instrument_app(app). Pass the current span's ids to capture_exception(trace_id=..., span_id=...) to find an error's trace in your tracing backend.

Retention and Partitions

tb_error_occurrence, tb_error_notification_log, otel_traces and otel_metrics are partitioned by month (<table>_YYYY_MM), so old rows are dropped a month at a time instead of deleted row by row:

-- Create the partitions of all four tables for this month and the next 3 (idempotent)
SELECT * FROM ensure_monitoring_partitions(3);

-- Drop the partitions older than 6 months
SELECT * FROM drop_old_error_occurrence_partitions(6);

-- Rows and size per partition
SELECT * FROM get_partition_stats();

Run the first two monthly, from cron or pg_cron:

0 0 1 * * psql -d myapp -c "SELECT ensure_monitoring_partitions(3);"
0 0 1 * * psql -d myapp -c "SELECT drop_old_error_occurrence_partitions(6);"

tb_error_log (one row per fingerprint) isn't partitioned; prune resolved errors yourself once their occurrences are gone:

DELETE FROM tb_error_log e
WHERE status = 'resolved'
  AND last_seen < NOW() - INTERVAL '6 months'
  AND NOT EXISTS (SELECT 1 FROM tb_error_occurrence o WHERE o.error_id = e.error_id);

Filter tb_error_occurrence queries on occurred_at so PostgreSQL scans only the partitions in range.

Dashboards

FraiseQL ships no Grafana dashboard for these tables. With PostgreSQL as a Grafana data source, the SQL above makes table and time-series panels; for the Prometheus metrics, see the runbooks' panels.

Next Steps