Mach5 - OpenTelemetry → AWS CloudWatch Logs

Setup Custom OTEL Configuration with Mach5 to enable AWS CloudWatch

This document describes how to configure OpenTelemetry (OTEL) to send Mach5 logs to AWS CloudWatch Logs using a custom OTEL configuration

Architecture Overview

Mach5 Pods
|
|  (OTLP logs)
v
OpenTelemetry Collector (Otel agent)
|
|  awscloudwatchlogs exporter
v
AWS CloudWatch Logs

Prerequisites

• Mach5 is deployed successfully

Custom OTEL Configuration

Full Custom Configuration

This custom otel configuration is defined as separate values.custom-otel-config.yaml

otel:
  custom:
    exporters:
      awscloudwatchlogs:
        region: us-east-1
        log_group_name: "/mach5/otel/logs"
        log_stream_name: "mach5-cluster"
        log_retention: 14
        tags:
          env: "dev"
          team: "mach5"

    processors:
      resource:
        attributes:
          - key: service.name
            value: "mach5"
            action: upsert
          - key: deployment.environment
            value: "dev"
            action: upsert

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [k8sattributes, resource, memory_limiter, batch]
          exporters: [awscloudwatchlogs]

Key Fields explanation

• exporters.awscloudwatchlogs

• processors.resource

Used to enrich every log record with consistent metadata.

Configuring the custom OTEL config

Upgrade Mach5 installation with the custom values file values.custom-otel-config.yaml for the custom OTEL config to take effect

helm upgrade --install <Installation-name> -n <NAMESPACE> mach5-search-<version>.tgz  \
-f /<path-to-local-mach5-values-file/>values.yaml \
-f values.custom-otel-config.yaml 

Once the upgrade is complete, you should be able to see the logs in CloudWatch

Verifying Logs in CloudWatch

• Open CloudWatch Logs

AWS Console → CloudWatch → Logs

• Locate Log Group

/mach5/otel/logs

• Open Log Stream

mach5-cluster

• Validate Log Content

Cloudwatch logs should have following:

• Mach5 log messages
• Kubernetes metadata fields
• service.name = mach5
• deployment.environment = dev

Troubleshooting - Enable Debug Exporter

For the debug exporter to take effect, you would need to run the above upgrade command again.

exporters:
  debug:
    verbosity: detailed

Pipeline:

exporters: [debug, awscloudwatchlogs]

This prints logs before export, confirming OTEL ingestion.