Enabling Tracing on a deployed OpenRelik Server

This guide outlines the steps to enable trace collection on your OpenRelik server.

1. Choose an OTel Receiver

In order to colelct the OpenTelemetry traces generated by OpenRelik server and other components, you need to have a running OpenTelemetry compatible Receiver.

OpenRelik has been tested with:

• Jaeger
• Google Cloud Trace API

If you decide to run your own Jaeger instance, here’s a very simple, not secure and not production-ready, docker-compose.yml example file you can use for testing:

services:
  jaeger-server:
    container_name: jaeger
    image: jaegertracing/jaeger:latest
    restart: always
    depends_on:
      - opensearch-jaeger-storage
    ports:
      - "127.0.0.1:16686:16686"
      - "127.0.0.1:4317:4317"
      - "127.0.0.1:4318:4318"
      - "127.0.0.1:5778:5778"
      - "127.0.0.1:9411:9411"
    volumes:
      - config.yaml:/jaeger/config.yaml
    environment:
      QUERY_BASE_PATH: /jaeger
    command:
      - "--config"
      - "/jaeger/config.yaml"
      - "--set"
      - "extensions.jaeger_query.base_path=/jaeger"

  opensearch-jaeger-storage:
    container_name: opensearch-jaeger-storage
    image: opensearchproject/opensearch:2.15.0
    ports:
      - "127.0.0.1:9200:9200"
    environment:
      - discovery.type=single-node
      - "plugins.security.disabled=true"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
    volumes:
      - ./data/opensearch:/usr/share/opensearch/data/

You can get an example config.yaml file, from the Jaeger’s project example.

2. Enable Telemetry

Per the code in setup_telemetry(), enabling traces is done by setting an environment variable OPENRELIK_OTEL_MODE to a string value that starts with "otlp-":

• otlp-grpc to generate traces to an OTel receiver over GRPC (for example: Jaeger)
• otlp-http to generate traces to an OTel receiver over HTTP (for example: Jaeger)
• otel-default-gce to generate traces to the Google Cloud Traces API, when running on an GCE VM (using the VM’s credentials)

otel-default-gce is the recommended way, as the other setups described above don’t include any encryption layer or authentication.

If you’re setting either otel-grpc or otel-http, you also need to specify the relevant endpoint address using either OPENRELIK_OTLP_GRPC_ENDPOINT or OPENRELIK_OTLP_HTTP_ENDPOINT.

These environment variables need to be set for every container that would generate traces.

To do this, at the top of your docker-compose.yml file, add, for example, to generate trace over GRPC to a Jaeger container named jaeger:

x-telemetry: &telemetry
  OPENRELIK_OTEL_MODE: otlp-grpc
  OPENRELIK_OTLP_GRPC_ENDPOINT: jaeger:4317

Or in the case of using GCP Cloud Trace API:

x-telemetry: &telemetry
  OPENRELIK_OTEL_MODE: otlp-default-gce

And in each of the relevant service, add <<: *telemetry to the environment sectrion.

For example, for the openrelik-worker-strings container, it becomes:

  openrelik-worker-strings:
    container_name: openrelik-worker-strings
    image: ghcr.io/openrelik/openrelik-worker-strings:${OPENRELIK_WORKER_STRINGS_VERSION}
    restart: always
    environment:
      <<: *telemetry
      REDIS_URL: redis://openrelik-redis:6379
    volumes:
      - ./data:/usr/share/openrelik/data
    command: "celery --app=src.app worker --task-events --concurrency=4 -Q openrelik-worker-strings"

** Example: Instrument OpenRelik worker code**

The first step is to call telemetry.setup_telemetry() from the Openrelik-Common library, and then telemetry.instrument_celery_app() in src/app.py.

Then traces should be automatically generated and appended to the proper spans. You can then use different helper methods in the Openrelik-Common telemetry.py file to add metadata to the traces.

In src/app.py:

   from openrelik_common import telemetry

   telemetry.setup_telemetry('openrelik-worker-strings')

   celery = Celery(...)

   telemetry.instrument_celery_app(celery)

In src/tasks.py:

   from openrelik_comon import telemetry

   @celery.task(bind=True, name=TASK_NAME, metadata=TASK_METADATA)
   def strings(...):

     <...>

     telemetry.add_attribute_to_current_span("task_config", task_config)


**Note:**

* Adjust the configuration to suit your specific needs and monitoring requirements.