Mach5 Telemetry: Metrics, Traces, and Logs

Mach5 emits telemetry through OpenTelemetry (OTel). This page explains the default in-cluster telemetry path, how to scrape Mach5 metrics with Prometheus, and how to add custom OTel exporters for traces and logs.

This page applies to Mach5 Enterprise deployments installed with the Mach5 Helm chart.

Telemetry architecture

Mach5 includes an in-cluster OpenTelemetry pipeline for operational telemetry.

At a high level:

Mach5 workloads
        |
        | OpenTelemetry telemetry
        v
Mach5 OTel agent
        |
        | metrics
        v
Mach5 metrics-server
        |
        | Prometheus-compatible /metrics endpoint
        v
Prometheus or another metrics scraper

The administrator-facing surfaces are:

By default, Mach5 exposes metrics through the metrics-server. Traces and logs are collected by the in-cluster OpenTelemetry agent, but they are not sent to an external backend unless you configure one.

Metrics with Prometheus

There are two related but separate uses of “Prometheus” in Mach5 telemetry:

1. Prometheus-compatible metrics format
   Mach5 exposes metrics in a format that Prometheus and compatible scrapers can read.

2. A Prometheus server deployment
   The Mach5 Helm chart can optionally deploy Prometheus for you.

Mach5 exposes the scrape endpoint regardless of whether the bundled Prometheus server is enabled:

http://<release>-metrics-server.<namespace>.svc.cluster.local:8080/metrics

Use:

• prometheus.enabled: true if you want the Mach5 Helm chart to deploy Prometheus.
• prometheus.enabled: false if your cluster already has Prometheus, Grafana Agent, an OpenTelemetry Collector, or another scraper.

In both cases, the metrics endpoint remains available at <release>-metrics-server:8080/metrics.

Validate the metrics endpoint

Port-forward the metrics-server service and fetch /metrics:

kubectl port-forward -n <namespace> svc/<release>-metrics-server 8080:8080

In another terminal:

curl http://127.0.0.1:8080/metrics

A successful response contains Prometheus-format metrics.

Custom OTel pipelines

Use custom OpenTelemetry configuration when you want Mach5 telemetry to go to an external backend, such as a tracing system, log platform, or vendor-managed collector.

Mach5 uses the OpenTelemetry Collector Contrib distribution. For more detailed configuration reference, see:

• OpenTelemetry Collector for the general Collector configuration model.
• OpenTelemetry Collector Contrib for contrib components such as additional exporters, receivers, processors, and extensions.

The collector image version is pinned by the Helm chart value otel.image. When copying examples from upstream documentation, check the Collector version in your chart and confirm that the component and options you use are supported by that version.

Most customizations follow this pattern:

1. Add a new exporter under otel.custom.exporters.
2. Update the relevant pipeline to use that exporter.

For example, this sends traces to an external OTLP collector:

otel:
  custom:
    exporters:
      otlp/traces:
        endpoint: "otel-collector.observability.svc.cluster.local:4317"
        tls:
          insecure: true
        # headers:
        #   api-key: "<token>"

    service:
      pipelines:
        traces:
          exporters: [otlp/traces]

Mach5 already provides the default receivers and processors for the built-in OTel agent. In many cases, you only need to change the exporter list for the pipeline you care about.

One important detail: list values are replaced, not appended. If you override a pipeline’s exporters list, include every exporter you still want for that pipeline.

For example, if you customize the metrics pipeline and still want Mach5 metrics to appear at <release>-metrics-server:8080/metrics, keep otlp/metrics in the list:

otel:
  custom:
    service:
      pipelines:
        metrics:
          exporters: [otlp/metrics, otlp/vendor]

Omit otlp/metrics only if you intentionally want to stop sending metrics to the Mach5 metrics-server.

Apply the values with your normal Mach5 Helm upgrade command:

helm upgrade --install <release> -n <namespace> mach5-search-<version>.tgz \
  -f values.yaml \
  -f values.telemetry.yaml

Recipes

AWS CloudWatch Logs

Mach5 also has a provider-specific guide for exporting logs to AWS CloudWatch Logs.

See Custom OTEL to AWS CloudWatch.