Observability¶
FraiseQL has three observability pieces. None is on by default; each is set up in your application code:
| Piece | API | Where the data goes |
|---|---|---|
| Error tracking | fraiseql.monitoring.init_error_tracker |
PostgreSQL tables tb_error_log, tb_error_occurrence |
| Metrics | fraiseql.monitoring.setup_metrics |
A Prometheus /metrics endpoint |
| Tracing | fraiseql.tracing.setup_tracing |
An OpenTelemetry exporter (OTLP, Jaeger or Zipkin) |
Errors live in your database, so you can query them with SQL and join them with your own tables. Metrics and traces go to the tools built for them (Prometheus, an OpenTelemetry collector).
Error Tracking¶
PostgreSQLErrorTracker stores exceptions in PostgreSQL, grouped by fingerprint
(like Sentry issues), with every occurrence kept in a partitioned table.
Schema¶
The tables, views and functions are in schema.sql, in the installed
fraiseql.monitoring package. FraiseQL doesn't create them: run the file once per
database, as part of your migrations.
SCHEMA=$(python -c "import fraiseql.monitoring, pathlib; print(pathlib.Path(fraiseql.monitoring.__file__).parent / 'schema.sql')")
psql "$DATABASE_URL" -f "$SCHEMA"
It creates:
| Object | What it holds |
|---|---|
tb_error_log |
One row per fingerprint: error_type, error_message, stack_trace, request_context / user_context / application_context (JSONB), severity, tags, environment, release_version, first_seen, last_seen, occurrence_count, status (unresolved, resolved, ignored, investigating), assigned_to, resolved_at, resolved_by, trace_id, span_id |
tb_error_occurrence |
One row per capture (error_id, occurred_at, contexts, stack_trace, breadcrumbs, trace_id), partitioned by month on occurred_at |
tb_error_notification_config, tb_error_notification_log |
Notification rules and their delivery log (see Notifications) |
v_active_errors, v_top_errors, v_error_trends |
Unresolved errors; the most frequent ones; errors per hour |
get_error_stats(p_hours), resolve_error(p_error_id, p_resolved_by, p_resolution_notes) |
SQL counterparts of the tracker methods |
ensure_monitoring_partitions(months_ahead), create_monthly_partition(table, date), create_error_occurrence_partition(date), ensure_error_occurrence_partitions(months_ahead), drop_old_error_occurrence_partitions(retention_months), get_partition_stats() |
Partition management (see Retention) |
otel_traces, otel_metrics |
Tables for spans and metrics, partitioned by month. Nothing in FraiseQL writes to them: traces and metrics go to the exporters below |
The file creates the partitions of the four partitioned tables (tb_error_occurrence,
tb_error_notification_log, otel_traces, otel_metrics) for the current month and
the next two. A capture fails (and is logged, not raised) when its month has no
partition, so create them ahead of time (see Retention).
Setup¶
from psycopg_pool import AsyncConnectionPool
from fraiseql.monitoring import init_error_tracker
pool = AsyncConnectionPool("postgresql://localhost/mydb", open=False)
async def startup() -> None:
await pool.open()
init_error_tracker(
pool,
environment="production",
release_version="2.4.1",
enable_notifications=False, # see Notifications below
)
init_error_tracker sets the process-wide tracker that get_error_tracker()
returns, and returns it.
Capturing Errors¶
FraiseQL captures nothing on its own: call capture_exception where you handle
errors. A resolver's exception becomes a GraphQL error inside the response and
never reaches an HTTP middleware, so capture it in the resolver:
import fraiseql
from fraiseql.monitoring import get_error_tracker
@fraiseql.query
async def invoice(info, id: int) -> Invoice | None:
try:
return await info.context["db"].find_one("v_invoice", mandatory_filters={"id": id})
except Exception as error:
tracker = get_error_tracker()
if tracker is not None:
await tracker.capture_exception(
error,
context={
"request": {"operation": "invoice", "invoice_id": id},
"user": {"user_id": str(info.context.get("user_id"))},
},
severity="error",
tags=["billing"],
)
raise
capture_exception(error, context=None, severity="error", tags=None, trace_id=None,
span_id=None) returns the error's id. The context keys it reads are request,
user and application (stored in the matching JSONB columns), breadcrumbs
(stored with the occurrence) and fingerprint (a string that replaces the default
grouping). capture_message(message, level="info", context=None, tags=None) records
a message the same way.
Errors are grouped by a fingerprint of the exception type and the file, line and
function where it was raised (the last frame outside FraiseQL and site-packages).
A new occurrence of a known fingerprint increments occurrence_count, updates
last_seen, and re-opens a resolved error.
Reading and Resolving Errors¶
tracker = get_error_tracker()
errors = await tracker.get_unresolved_errors(limit=20, severity="error")
details = await tracker.get_error(errors[0]["error_id"])
stats = await tracker.get_error_stats(hours=24)
# {'total_errors': ..., 'unresolved_errors': ..., 'unique_error_types': ..., 'avg_resolution_time_hours': ...}
await tracker.resolve_error(errors[0]["error_id"], resolved_by="alice", resolution_notes="Fixed in 2.4.2")
Or in SQL:
-- Most frequent unresolved errors
SELECT error_type, error_message, occurrence_count, last_seen
FROM v_top_errors
LIMIT 10;
-- Errors of the last 24 hours, with the users they affected
SELECT
e.error_type,
e.error_message,
COUNT(*) AS occurrences,
COUNT(DISTINCT o.user_context->>'user_id') AS affected_users,
MAX(o.occurred_at) AS last_seen
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
WHERE o.occurred_at > NOW() - INTERVAL '24 hours'
GROUP BY e.error_id, e.error_type, e.error_message
ORDER BY occurrences DESC
LIMIT 20;
-- Occurrences per hour over the last week (filter on occurred_at to prune partitions)
SELECT date_trunc('hour', o.occurred_at) AS hour, e.error_type, COUNT(*) AS count
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
WHERE o.occurred_at > NOW() - INTERVAL '7 days'
GROUP BY hour, e.error_type
ORDER BY hour DESC, count DESC;
-- Errors by release
SELECT release_version, COUNT(*) AS errors, SUM(occurrence_count) AS occurrences
FROM tb_error_log
GROUP BY release_version
ORDER BY occurrences DESC;
Because the errors are rows in your database, you can join them with your own tables, for example on an id you put in the context:
SELECT e.error_type, e.error_message, o.occurred_at, i.amount
FROM tb_error_occurrence o
JOIN tb_error_log e ON e.error_id = o.error_id
JOIN tb_invoice i ON i.id = (o.request_context->>'invoice_id')::int
WHERE o.occurred_at > NOW() - INTERVAL '24 hours';
Notifications¶
The tracker sends a notification when it captures an error
(enable_notifications=True, the default), in a background task: its
NotificationManager reads the enabled rules of tb_error_notification_config that
match the error (error_fingerprint, error_type, severity and environment
filters, NULL matching everything, and min_occurrence_count), skips a rule that
already sent for this error within its rate_limit_minutes, sends through the rule's
channel_type (email, slack, webhook, or one added with register_channel),
and logs the attempt (sent or failed, with the reason) in
tb_error_notification_log.
INSERT INTO tb_error_notification_config
(error_type, severity, environment, channel_type, channel_config, rate_limit_minutes)
VALUES
(NULL, ARRAY['error', 'critical'], ARRAY['production'], 'webhook',
'{"url": "https://hooks.example.com/errors", "headers": {"Authorization": "Bearer ..."}}', 30),
('PaymentError', NULL, NULL, 'slack',
'{"webhook_url": "https://hooks.slack.com/services/...", "channel": "#payments"}', 60),
(NULL, ARRAY['critical'], NULL, 'email',
'{"smtp_host": "smtp.example.com", "smtp_user": "alerts", "smtp_password": "...",
"from_address": "alerts@example.com", "to": ["oncall@example.com"]}', 15);
channel_config is passed to the channel's send; a channel class also gets the
keys its constructor takes (EmailChannel its smtp_host, smtp_port, smtp_user,
smtp_password, use_tls and from_address). For a channel of your own, register
it on a manager and give that manager to the tracker:
from fraiseql.monitoring.notifications import NotificationManager
from fraiseql.monitoring.postgres_error_tracker import init_error_tracker
class PagerChannel:
async def send(self, error: dict, config: dict) -> tuple[bool, str | None]:
await page(config["service"], f"{error['error_type']}: {error['error_message']}")
return True, None # (sent?, error message)
def format_message(self, error: dict, template: str | None = None) -> str:
return error["error_message"]
manager = NotificationManager(pool)
manager.register_channel("pager", PagerChannel) # a class, or an instance
tracker = init_error_tracker(pool, environment="production", notification_manager=manager)
tb_error_notification_log, like tb_error_occurrence, otel_traces and
otel_metrics, is partitioned by month: schema.sql creates the partitions of the
current month and the next two. Run SELECT ensure_monitoring_partitions(2); every
month (pg_cron or any scheduler), or writes fail once the months run out.
Metrics¶
setup_metrics(app) adds a middleware that counts and times every HTTP request, and
a GET /metrics endpoint for Prometheus (it needs prometheus_client):
from fraiseql.fastapi import create_fraiseql_app
from fraiseql.monitoring import MetricsConfig, setup_metrics
app = create_fraiseql_app(database_url="postgresql://localhost/mydb", types=[User], queries=[users])
metrics = setup_metrics(app, MetricsConfig(namespace="fraiseql"))
The metrics recorded on their own are fraiseql_http_requests_total,
fraiseql_http_request_duration_seconds, fraiseql_response_time_seconds and
fraiseql_errors_total (exceptions that escape the app). The GraphQL, database
pool and cache metrics only get values your code records (@with_metrics,
metrics.update_db_connections(...), ...). The full list, and the queries to use
them, are in the runbooks' metrics reference.
Distributed Tracing¶
With the tracing extra (pip install "fraiseql[tracing]": the OpenTelemetry SDK,
the OTLP exporter and the psycopg instrumentation),
fraiseql.tracing.setup_tracing(app, TracingConfig(...)) opens a span per HTTP
request (POST /graphql, kind SERVER, with the http.* attributes and the status
code; exclude_paths defaults to /health, /ready, /metrics, /docs,
/openapi.json), continues the trace of an incoming traceparent header, instruments
psycopg, and exports the spans:
from fraiseql.fastapi import create_fraiseql_app
from fraiseql.tracing import TracingConfig, setup_tracing
app = create_fraiseql_app(database_url="postgresql://localhost/mydb", types=[User], queries=[users])
setup_tracing(
app,
TracingConfig(
service_name="orders-api",
deployment_environment="production",
sample_rate=0.1, # keep 10% of traces
export_format="otlp", # or "console", "zipkin"
export_endpoint="http://otel-collector:4317", # OTLP over gRPC
),
)
export_format="console" prints the spans (no endpoint needed); "zipkin" needs
opentelemetry-exporter-zipkin-json. "jaeger" raises an ImportError: its
exporter is deprecated upstream and doesn't install with the current SDK; Jaeger
accepts OTLP, so use "otlp" with its port 4317. Without export_endpoint (OTLP,
Zipkin), spans are recorded but not exported. If a TracerProvider is already set
when setup_tracing runs, fraiseql uses it as it is (its exporters, not the config's).
Or set up OpenTelemetry yourself; this traces every SQL statement FraiseQL runs, and spans you open in resolvers:
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.psycopg import PsycopgInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
import fraiseql
provider = TracerProvider(
resource=Resource.create({"service.name": "orders-api", "deployment.environment": "production"}),
sampler=TraceIdRatioBased(0.1), # keep 10% of traces
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)))
trace.set_tracer_provider(provider)
PsycopgInstrumentor().instrument() # before the app opens its pool
tracer = trace.get_tracer("orders-api")
@fraiseql.query
async def users(info) -> list[User]:
with tracer.start_as_current_span("users"):
return await info.context["db"].find("v_user")
For a span per HTTP request, add opentelemetry-instrumentation-fastapi and call
FastAPIInstrumentor.instrument_app(app). Pass the current span's ids to
capture_exception(trace_id=..., span_id=...) to find an error's trace in your
tracing backend.
Retention and Partitions¶
tb_error_occurrence, tb_error_notification_log, otel_traces and otel_metrics
are partitioned by month (<table>_YYYY_MM), so old rows are dropped a month at a
time instead of deleted row by row:
-- Create the partitions of all four tables for this month and the next 3 (idempotent)
SELECT * FROM ensure_monitoring_partitions(3);
-- Drop the partitions older than 6 months
SELECT * FROM drop_old_error_occurrence_partitions(6);
-- Rows and size per partition
SELECT * FROM get_partition_stats();
Run the first two monthly, from cron or pg_cron:
0 0 1 * * psql -d myapp -c "SELECT ensure_monitoring_partitions(3);"
0 0 1 * * psql -d myapp -c "SELECT drop_old_error_occurrence_partitions(6);"
tb_error_log (one row per fingerprint) isn't partitioned; prune resolved errors
yourself once their occurrences are gone:
DELETE FROM tb_error_log e
WHERE status = 'resolved'
AND last_seen < NOW() - INTERVAL '6 months'
AND NOT EXISTS (SELECT 1 FROM tb_error_occurrence o WHERE o.error_id = e.error_id);
Filter tb_error_occurrence queries on occurred_at so PostgreSQL scans only the
partitions in range.
Dashboards¶
FraiseQL ships no Grafana dashboard for these tables. With PostgreSQL as a Grafana data source, the SQL above makes table and time-series panels; for the Prometheus metrics, see the runbooks' panels.
Next Steps¶
- Monitoring Guide - Detailed monitoring setup
- Runbooks - Incident procedures and Prometheus queries
- Deployment - Production deployment patterns
- Health Checks - Application health monitoring