Skip to content
Distributed Tracing
On this page

Distributed Tracing

Pipelines-as-Code emits trace spans for webhook event processing and PipelineRun lifecycle timing.

Enabling tracing

Two configuration paths can enable tracing.

Via OpenTelemetry environment variables

Set on the controller and watcher pods:

  • OTEL_EXPORTER_OTLP_ENDPOINT - OTLP collector endpoint URL. Required.
  • OTEL_TRACES_SAMPLER - Sampler family. Required. Supported: always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio.
  • OTEL_TRACES_SAMPLER_ARG - Numeric argument for ratio samplers. Example: 0.1 with parentbased_traceidratio samples 10% of root traces while keeping the chain coherent.
  • OTEL_EXPORTER_OTLP_PROTOCOL (or traces-specific OTEL_EXPORTER_OTLP_TRACES_PROTOCOL) - OTLP transport: grpc or http/protobuf. Default: grpc.

Both OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_TRACES_SAMPLER must be set. Inbound traceparent headers on webhook requests are honored via the W3C TraceContext propagator. Changes take effect on the next pod restart.

Sampler choice and chain coherency

The parentbased_* sampler family honors the parent span’s sample decision carried in the W3C traceparent flag bit. When every service in the delivery chain uses parent-based samplers, the root span’s sampling decision propagates end to end: each service either keeps its spans or drops them based on what the root chose. Flat-rate samplers (traceidratio without parent-based) cause each service to roll independently, which at fractional sampling fragments the chain into orphaned spans whose parent_spanID references a span that was dropped. parentbased_always_on keeps everything; parentbased_traceidratio with a numeric argument samples a coherent fraction.

Via Knative observability ConfigMap

Set in pipelines-as-code-config-observability:

  • tracing-protocol - grpc, http/protobuf, stdout, or none. Default: none.
  • tracing-endpoint - Collector endpoint for grpc or http/protobuf.
  • tracing-sampling-rate - Sample fraction. Per-component independent.

Changes to Knative’s tracing config require restarting the controller and watcher pods. The tracer is built once at startup.

When both are configured

OpenTelemetry takes precedence when set: all spans flow through the OpenTelemetry exporter. With OpenTelemetry unset, Knative’s tracer (from the ConfigMap above) remains as the runtime tracer.

To use only OpenTelemetry, set tracing-protocol: none in pipelines-as-code-config-observability.

To use only Knative, unset OTEL_EXPORTER_OTLP_ENDPOINT on the controller and watcher pods.

Emitted spans

The controller emits a PipelinesAsCode:ProcessEvent span for each webhook event. The watcher emits waitDuration and executeDuration spans for completed PipelineRuns. The OTel resource attribute service.name on all emitted spans is pipelines-as-code.

Webhook event span (PipelinesAsCode:ProcessEvent)

OTel VCS semantic conventions:

Attribute Source
vcs.provider.name Git provider name
vcs.repository.url.full Repository URL
vcs.ref.head.revision Head commit SHA

PaC-specific:

Attribute Source
pipelinesascode.tekton.dev.event_type Webhook event type

PipelineRun timing spans (waitDuration, executeDuration)

Tekton-compatible bare keys (match Tekton’s own reconciler spans for correlation):

Attribute Source
namespace PipelineRun namespace
pipelinerun PipelineRun name

Cross-service delivery attributes (delivery.tekton.dev.*):

Attribute Source
delivery.tekton.dev.pipelinerun_uid PipelineRun UID
delivery.tekton.dev.result_message First failing TaskRun message; omitted on success; truncated to 1024 bytes

Additional delivery.tekton.dev.* attributes are sourced from configurable PipelineRun labels.

OTel CI/CD semantic conventions (executeDuration only):

Attribute Source
cicd.pipeline.result Outcome enum (see below)

cicd.pipeline.result enum

Condition Value
Status=True success
Status=False, reason Failed failure
Status=False, reason PipelineRunTimeout timeout
Status=False, reason Cancelled or CancelledRunningFinally cancellation
Status=False, any other reason error

Configuring label-sourced attributes

Some span attributes are read from PipelineRun labels. The label names are configurable via the main pipelines-as-code ConfigMap so deployments can point at their existing labels without rewriting producers:

ConfigMap key PipelineRun label read (default) Span attribute emitted
tracing-label-action delivery.tekton.dev/action cicd.pipeline.action.name
tracing-label-application delivery.tekton.dev/application delivery.tekton.dev.application
tracing-label-component delivery.tekton.dev/component delivery.tekton.dev.component

Setting a ConfigMap key to the empty string disables emission of that label-sourced attribute. Only label-sourced attributes are affected; all other span attributes are always emitted. The emitted span attribute keys are fixed regardless of which labels are read, so cross-service queries work uniformly.

Unlike the observability ConfigMap above (which requires a pod restart), changes to these label mappings are picked up automatically without restarting pods.

Trace context propagation

When Pipelines-as-Code creates a PipelineRun, it sets the tekton.dev/pipelinerunSpanContext annotation with a JSON-encoded OTel TextMapCarrier containing the W3C traceparent. PaC tracing works independently - you get PaC spans regardless of whether Tekton Pipelines has tracing enabled.

If Tekton Pipelines is also configured with tracing pointing at the same collector, its reconciler spans appear as children of the PaC span, providing a single end-to-end trace from webhook receipt through task execution. See the Tekton Pipelines tracing documentation for Tekton’s independent tracing setup.

Deploying a trace collector

Pipelines-as-Code exports traces using the standard OpenTelemetry Protocol (OTLP). You need a running OTLP-compatible collector for OTEL_EXPORTER_OTLP_ENDPOINT to point to. Common options include:

Deploying and operating a collector is outside the scope of Pipelines-as-Code. Refer to your organization’s observability infrastructure or the links above for setup instructions.