Getting Started with Prometheus

Access Prometheus Service

1. Log into the Thalassa Cloud Console
2. Navigate to Observability → Prometheus
3. Click Create Tenant

Configure Tenant

1. When creating your Prometheus tenant, choose a descriptive name that reflects its purpose, such as production-monitoring or infrastructure-observability.
2. The tenant will serve as the target for your remote write data and querying. Select the region where you want your tenant to be hosted, which should typically match the region where your infrastructure is located.
3. Configure the metrics retention period based on your compliance and analysis requirements; common retention periods range from 30 days for operational monitoring to 90 days or longer for compliance and capacity planning.
4. Once you’ve configured these settings, click Create to create the tenant.

Retrieve Remote Write Endpoint

1. Navigate to your Prometheus tenant in the console
2. Go to Configuration → Remote Write
3. Copy the Remote Write URL (e.g., https://prometheus.nl-01.thalassa.cloud/api/v1/push)

Get Authentication Credentials

Thalassa Prometheus Service uses OIDC authentication integrated with Thalassa Cloud IAM. For production deployments and automation, we recommend using a service account. To create a service account, navigate to IAM → Service Accounts in the console, create a new service account, and grant it appropriate permissions for Prometheus remote write. Once created, generate access credentials for the service account, which you’ll use in your Prometheus configuration.

Alternatively, you can use a personal access token for authentication. Create a personal access token in your account settings and use it in your Prometheus configuration. Note that personal access tokens are less suitable for automated deployments and should be used primarily for testing or manual operations.

Configure IP ACLs (Optional)

Optionally, you can restrict remote write access by IP address. Navigate to Configuration → Remote Write → IP ACLs in your Prometheus tenant settings. Add the IP addresses or CIDR blocks that should be allowed to send metrics to the remote write endpoint. Once configured, only requests originating from these IP addresses will be accepted, providing an additional layer of security for your metrics ingestion.

Update Prometheus Configuration

Edit your Prometheus configuration file (prometheus.yml):

global:
  scrape_interval: 15s
  external_labels:
    cluster: production
    region: nl-01

# Your scrape configs
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

# Remote write configuration
remote_write:
  - url: https://prometheus.nl-01.thalassa.cloud/api/v1/push
    headers:
      x-org-id: "<your-tenant-id>"
    oauth2:
      client_id: <client-credenital-key>
      client_secret: <service-account-secret>
      token_url: https://api.thalassa.cloud/v1/oidc/token
    queue_config:
      max_samples_per_send: 1000
      batch_send_deadline: 5s
      max_retries: 3
    write_relabel_configs:
      # Optional: filter metrics before sending
      - source_labels: [__name__]
        regex: 'up|prometheus_.*'
        action: drop

Using OIDC Token Exchange

For CI/CD or automated deployments, use OIDC token exchange:

# Exchange OIDC token for Thalassa Cloud access token
TOKEN=$(tcloud oidc token-exchange \
  --subject-token "${OIDC_TOKEN}" \
  --organisation-id "${THALASSA_ORGANISATION_ID}" \
  --service-account-id "${THALASSA_SERVICE_ACCOUNT_ID}")

# Use token in Prometheus remote_write config

Reload Prometheus

Reload your Prometheus configuration:

# If using systemd
sudo systemctl reload prometheus

# Or send SIGHUP
kill -HUP $(pgrep prometheus)

# Or use Prometheus API
curl -X POST http://localhost:9090/-/reload

Check Remote Write Status

1. Access your Prometheus UI (typically http://localhost:9090)
2. Navigate to Status → Targets to verify scrape targets are working
3. Go to Status → Runtime & Build Information → Remote Write
4. Verify remote write endpoints show as “UP”

Query Metrics

Use the Thalassa Prometheus Query API to verify metrics are being stored:

# Query metrics using the Query API
curl -H "Authorization: Bearer $TOKEN" -H "x-org-id: $TENANT_ID" \
  "https://prometheus.nl-01.thalassa.cloud/api/v1/query?query=up"

Or use the console:

1. Navigate to your Prometheus tenant
2. Go to Query or access the Query API
3. Run a simple query: up
4. Verify you see metrics from your Prometheus instance

Create Alert Rule

You can create alert rules using the console, Mimirtool, or Prometheus APIs. Here’s how to do it via the console:

1. Navigate to your Prometheus tenant
2. Go to Alerting → Alert Rules
3. Click Create Alert Rule
4. Configure a basic alert:

Example: High CPU Usage

groups:
- name: infrastructure
  interval: 30s
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"
      description: "CPU usage is {{ $value }}% on {{ $labels.instance }}"

Configure Notification Channels

Navigate to Alerting → Notification Channels to configure how alerts are delivered. The service supports multiple notification channel types: SMTP for email notifications using a remote SMTP server, webhooks for sending alerts to custom endpoints, Slack integration for posting alerts to Slack workspaces, and Microsoft Teams integration for Teams notifications. After adding your notification channels, configure routing rules to determine which alerts are sent to which channels based on alert labels and severity levels.