Best Practices

Observability best practices

Use these practices to keep Observability workspaces reliable and cost-efficient.

Workspace design

Practice	Guidance
Scope	Prefer one workspace per environment or team boundary that should share query and alert scope
Region	Create the workspace in the same region as the collectors to reduce latency and data-transfer complexity
Retention	Set `retentionDays` deliberately; it applies to metrics and logs together
Credentials	Dedicated service accounts per environment; least-privilege Prometheus/Loki policies

Avoid putting unrelated production and noisy development traffic in the same workspace if you need separate retention, IAM, or cost attribution.

Label strategy (metrics)

Use stable labels: env, app, team, cluster, region.
Do not use user IDs, request IDs, or unbounded paths as label values.
Drop unused or high-cardinality series in the collector with write_relabel_configs before remote write.
Prefer recording rules for expensive aggregations instead of repeating heavy queries in dashboards.

Log stream labels

Keep Loki stream labels low-cardinality (app, namespace, level).
Put variable content in the log line, not in labels.
Align log and metric label names where it helps correlation, without copying high-cardinality metric labels onto streams.

Remote write and push volume

Because pricing tracks ingested volume, reduce samples and log lines you do not need:

Longer scrape intervals for non-critical metrics (30–60s).
Drop _bucket histograms you never query, or keep only selected histograms.
Filter debug-level logs in the agent for production workspaces.
Monitor remote-write queue size and Loki push errors after configuration changes.

Example metric filter:

remote_write:
  - url: https://prometheus.<region>.thalassa.cloud/workspace/obsw-<id>/api/v1/push
    write_relabel_configs:
      - source_labels: [__name__]
        regex: ".*_bucket"
        action: drop

Query efficiency

# Prefer filtering before aggregation
sum(rate(http_requests_total{app="backend"}[5m])) by (instance)

Use recording rules for dashboards that run the same heavy expression repeatedly. See Alerting for rule management.

Retention planning

Need	Typical retention
Incident response	7–30 days
Trends and capacity	30–90 days
Compliance archives	Up to 1095 days

Longer retention increases storage. Export or downsample externally if you need multi-year archives beyond what you want to keep hot in the workspace.

Security

Prefer service accounts over personal tokens for agents.
Separate ingest principals (remote write / push) from interactive query principals when possible.
Rotate client secrets on a schedule.
Treat workspace URLs as sensitive configuration — they embed the workspace identity.