Remote Write

Metrics remote write

Observability workspaces accept metrics through the Prometheus remote write protocol and OTLP metrics. Configure your collectors to use the workspace remote write URL from the console or API (remoteWriteUrl).

Basic Prometheus configuration

remote_write:
  - url: https://prometheus.<region>.thalassa.cloud/workspace/obsw-<id>/api/v1/push
    oauth2:
      client_id: <service-account-client-id>
      client_secret: <service-account-client-secret>
      token_url: https://api.thalassa.cloud/oidc/token

Use the exact URL for your workspace. Do not strip the /workspace/obsw-…/ path segment.

Prometheus Operator

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  remoteWrite:
    - url: https://prometheus.<region>.thalassa.cloud/workspace/obsw-<id>/api/v1/push
      oauth2:
        clientId:
          name: observability-credentials
          key: client-id
        clientSecret:
          name: observability-credentials
          key: client-secret
        tokenUrl: https://api.thalassa.cloud/oidc/token

Create the secret:

kubectl create secret generic observability-credentials \
  --from-literal=client-id=<service-account-client-id> \
  --from-literal=client-secret=<service-account-client-secret>

OTLP metrics

Workspaces also expose an OTLP metrics endpoint (remoteWriteOtlpUrl), typically:

https://prometheus.<region>.thalassa.cloud/workspace/obsw-<id>/api/v1/otlp/v1/metrics

Configure OpenTelemetry Collector (or compatible agents) with OIDC bearer authentication against Thalassa Cloud, the same way you authenticate remote write.

Authentication

Method	When to use
Service account OAuth2 client credentials	Production collectors and automation
Personal access token (Bearer)	Manual tests only
OIDC token exchange	CI/CD that already holds an external OIDC subject token

TOKEN=$(tcloud oidc token-exchange \
  --subject-token "${OIDC_TOKEN}" \
  --organisation-id "${THALASSA_ORGANISATION_ID}" \
  --service-account-id "${THALASSA_SERVICE_ACCOUNT_ID}")

The principal must be allowed to remote-write to the workspace (Prometheus data-plane IAM + token scopes). Workspace read-only control-plane access is not sufficient.

Write relabeling

Filter or transform series before they leave your collector:

remote_write:
  - url: https://prometheus.<region>.thalassa.cloud/workspace/obsw-<id>/api/v1/push
    oauth2:
      client_id: <service-account-client-id>
      client_secret: <service-account-client-secret>
      token_url: https://api.thalassa.cloud/oidc/token
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "up|prometheus_.*"
        action: drop
      - source_labels: [__name__]
        regex: "node_.*|http_.*"
        action: keep
      - target_label: environment
        replacement: production

Queue configuration

remote_write:
  - url: https://prometheus.<region>.thalassa.cloud/workspace/obsw-<id>/api/v1/push
    oauth2:
      client_id: <service-account-client-id>
      client_secret: <service-account-client-secret>
      token_url: https://api.thalassa.cloud/oidc/token
    queue_config:
      max_samples_per_send: 1000
      batch_send_deadline: 5s
      max_retries: 3
      min_backoff: 30ms
      max_backoff: 5s
      capacity: 10000

Tune batch size and capacity for your scrape volume. Prefer filtering high-cardinality series over enlarging queues indefinitely.

Troubleshooting

Symptom	Checks
401 / 403	Client credentials, token scopes, and Prometheus remote-write IAM on the workspace
404	Workspace ID and path; confirm status is `ready`
Growing remote-write queue	Network path to the region, rate of samples, and whether write relabeling drops enough series
Empty query results	Scrape targets healthy, remote write up, correct workspace, and retention window