Agent Configuration

SDK Autoconfiguration

The SDK’s autoconfiguration module is used for basic configuration of the agent. Read the docs to find settings such as configuring export or sampling.

Here are some quick links into those docs for the configuration options for specific portions of the SDK & agent:

Configuring the agent

The agent can consume configuration from one or more of the following sources (ordered from highest to lowest priority):

Configuration file

You can provide a path to agent configuration file by setting the following property:

System property: otel.javaagent.configuration-file Environment variable: OTEL_JAVAAGENT_CONFIGURATION_FILE

Description: Path to valid Java properties file which contains the agent configuration.

Extensions

You can enable extensions by setting the following property:

System property: otel.javaagent.extensions Environment variable: OTEL_JAVAAGENT_EXTENSIONS

Description: Path to an extension jar file or folder, containing jar files. If pointing to a folder, every jar file in that folder will be treated as separate, independent extension.

Common instrumentation configuration

Common settings that apply to multiple instrumentations at once.

Peer service name

The peer service name is the name of a remote service to which a connection is made. It corresponds to service.name in the resource for the local service.

System property: otel.instrumentation.common.peer-service-mapping Environment variable: OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING

Description: Used to specify a mapping from host names or IP addresses to peer services, as a comma-separated list of <host_or_ip>=<user_assigned_name> pairs. The peer service is added as an attribute to a span whose host or IP address match the mapping.

For example, if set to the following:

1.2.3.4=cats-service,dogs-abcdef123.serverlessapis.com=dogs-api

Then, requests to 1.2.3.4 will have a peer.service attribute of cats-service and requests to dogs-abcdef123.serverlessapis.com will have an attribute of dogs-api.

DB statement sanitization

The agent sanitizes all database queries/statements before setting the db.statement semantic attribute. All values (strings, numbers) in the query string are replaced with a question mark (?).

Examples:

  • SQL query SELECT a from b where password="secret" will appear as SELECT a from b where password=? in the exported span;
  • Redis command HSET map password "secret" will appear as HSET map password ? in the exported span.

This behavior is turned on by default for all database instrumentations. Use the following property to disable it:

System property: otel.instrumentation.common.db-statement-sanitizer.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED

Default: true
Description: Enables the DB statement sanitization.

Capturing HTTP request and response headers

You can configure the agent to capture predefined HTTP headers as span attributes, according to the semantic convention. Use the following properties to define which HTTP headers you want to capture:

System property: otel.instrumentation.http.capture-headers.client.request Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_REQUEST

Description: A comma-separated list of HTTP header names. HTTP client instrumentations will capture HTTP request header values for all configured header names.

System property: otel.instrumentation.http.capture-headers.client.response Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_CLIENT_RESPONSE

Description: A comma-separated list of HTTP header names. HTTP client instrumentations will capture HTTP response header values for all configured header names.

System property: otel.instrumentation.http.capture-headers.server.request Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST

Description: A comma-separated list of HTTP header names. HTTP server instrumentations will capture HTTP request header values for all configured header names.

System property: otel.instrumentation.http.capture-headers.server.response Environment variable: OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE

Description: A comma-separated list of HTTP header names. HTTP server instrumentations will capture HTTP response header values for all configured header names.

These configuration options are supported by all HTTP client and server instrumentations.

Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.

Capturing servlet request parameters

You can configure the agent to capture predefined HTTP request parameter as span attributes for requests that are handled by Servlet API. Use the following property to define which servlet request parameters you want to capture:

System property: otel.instrumentation.servlet.experimental.capture-request-parameters Environment variable: OTEL_INSTRUMENTATION_SERVLET_EXPERIMENTAL_CAPTURE_REQUEST_PARAMETERS

Description: A comma-separated list of request parameter names.

Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.

Capturing consumer message receive telemetry in messaging instrumentations

You can configure the agent to capture the consumer message receive telemetry in messaging instrumentation. Use the following property to enable it:

System property: otel.instrumentation.messaging.experimental.receive-telemetry.enabled Environment variable: OTEL_INSTRUMENTATION_MESSAGING_EXPERIMENTAL_RECEIVE_TELEMETRY_ENABLED

Default: false
Description: Enables the consumer message receive telemetry.

Note that this will cause the consumer side to start a new trace, with only a span link connecting it to the producer trace.

Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.

Suppressing specific auto-instrumentation

Disabling the agent entirely

You can disable the agent using -Dotel.javaagent.enabled=false (or using the equivalent environment variable OTEL_JAVAAGENT_ENABLED=false).

Suppressing specific agent instrumentation

You can suppress agent instrumentation of specific libraries by using -Dotel.instrumentation.[name].enabled=false (or using the equivalent environment variable OTEL_INSTRUMENTATION_[NAME]_ENABLED) where name (NAME) is the corresponding instrumentation name:

Library/FrameworkInstrumentation name
Additional methods tracingmethods
Additional tracing annotationsexternal-annotations
Akka Actorakka-actor
Akka HTTPakka-http
Apache Axis2axis2
Apache Camelapache-camel
Apache Cassandracassandra
Apache CXFcxf
Apache DBCPapache-dbcp
Apache Dubboapache-dubbo
Apache Geodegeode
Apache HttpAsyncClientapache-httpasyncclient
Apache HttpClientapache-httpclient
Apache Kafkakafka
Apache MyFacesmyfaces
Apache RocketMQrocketmq-client
Apache Struts 2struts
Apache Tapestrytapestry
Apache Tomcattomcat
Apache Wicketwicket
Armeriaarmeria
AsyncHttpClient (AHC)async-http-client
AWS Lambdaaws-lambda
AWS SDKaws-sdk
Azure SDKazure-core
Couchbasecouchbase
C3P0c3p0
Dropwizard Viewsdropwizard-views
Dropwizard Metricsdropwizard-metrics
Eclipse Grizzlygrizzly
Eclipse Jerseyjersey
Eclipse Jettyjetty
Eclipse Jetty HTTP Clientjetty-httpclient
Eclipse Metrometro
Eclipse Mojarramojarra
Eclipse Vert.x HttpClientvertx-http-client
Eclipse Vert.x Kafka Clientvertx-kafka-client
Eclipse Vert.x RxJavavertx-rx-java
Eclipse Vert.x Webvertx-web
Elasticsearch clientelasticsearch-transport
Elasticsearch REST clientelasticsearch-rest
Google Guavaguava
Google HTTP clientgoogle-http-client
Google Web Toolkitgwt
Grailsgrails
GraphQL Javagraphql-java
GRPCgrpc
Hibernatehibernate
HikariCPhikaricp
Java HTTP Clientjava-http-client
Java HttpURLConnectionhttp-url-connection
Java JDBCjdbc
Java JDBC DataSourcejdbc-datasource
Java RMIrmi
Java Servletservlet
java.util.concurrentexecutor
java.util.loggingjava-util-logging
JAX-RS (Client)jaxrs-client
JAX-RS (Server)jaxrs
JAX-WSjaxws
JBoss Logging Appenderjboss-logmanager-appender
JBoss Logging MDCjboss-logmanager-mdc
JMSjms
JSPjsp
K8s Clientkubernetes-client
kotlinx.coroutineskotlinx-coroutines
Log4j Appenderlog4j-appender
Log4j MDC (1.x)log4j-mdc
Log4j Context Data (2.x)log4j-context-data
Logback Appenderlogback-appender
Logback MDClogback-mdc
Micrometermicrometer
MongoDBmongo
Netflix Hystrixhystrix
Nettynetty
OkHttpokhttp
OpenLibertyliberty
OpenTelemetry Extension Annotationsopentelemetry-extension-annotations
OpenTelemetry Instrumentation Annotationsopentelemetry-instrumentation-annotations
OpenTelemetry APIopentelemetry-api
Oracle UCPoracle-ucp
OSHI (Operating System and Hardware Information)oshi
Play Frameworkplay
Play WS HTTP Clientplay-ws
Quartzquartz
RabbitMQ Clientrabbitmq
Ratpackratpack
ReactiveX RxJavarxjava
Reactorreactor
Reactor Nettyreactor-netty
Redis Jedisjedis
Redis Lettucelettuce
Rediscalarediscala
Redissonredisson
Restletrestlet
Scala ForkJoinPoolscala-fork-join
Spark Web Frameworkspark
Spring Batchspring-batch
Spring Corespring-core
Spring Dataspring-data
Spring JMSspring-jms
Spring Integrationspring-integration
Spring Kafkaspring-kafka
Spring RabbitMQspring-rabbit
Spring RMIspring-rmi
Spring Schedulingspring-scheduling
Spring Webspring-web
Spring WebFluxspring-webflux
Spring Web MVCspring-webmvc
Spring Web Servicesspring-ws
Spymemcachedspymemcached
Tomcat JDBCtomcat-jdbc
Twilio SDKtwilio
Twitter Finatrafinatra
Undertowundertow
Vaadinvaadin
Vibur DBCPvibur-dbcp

Note: When using environment variables, dashes (-) should be converted to underscores (_). For example, to suppress traces from akka-actor library, set OTEL_INSTRUMENTATION_AKKA_ACTOR_ENABLED to false.

Suppressing controller and/or view spans

Some instrumentations (e.g. Spring Web MVC instrumentation) produce SpanKind.Internal spans to capture the controller and/or view execution. These spans can be suppressed using the configuration settings below, without suppressing the entire instrumentation which would also disable the instrumentation’s capturing of http.route and associated span name on the parent SpanKind.Server span.

System property: otel.instrumentation.common.experimental.controller-telemetry.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_CONTROLLER_TELEMETRY_ENABLED

Default: true
Description: Enables the controller telemetry.

System property: otel.instrumentation.common.experimental.view-telemetry.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED

Default: true
Description: Enables the view telemetry.

Enable manual instrumentation only

You can suppress all auto instrumentations but have support for manual instrumentation with @WithSpan and normal API interactions by using -Dotel.instrumentation.common.default-enabled=false -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-annotations.enabled=true

Instrumentation span suppression behavior

Some of the libraries that this agent instruments in turn use lower-level libraries, that are also instrumented. This results in nested CLIENT spans (a span with the kind CLIENT has a child span with the same kind CLIENT). For example spans produced by Reactor Netty instrumentation will have children spans produced by Netty instrumentation. Or Dynamo DB spans produced by AWS SDK instrumentation will have children spans produced by http protocol library instrumentation.

By default the agent will suppress nested CLIENT spans for the same semantic convention.

By setting -Dotel.instrumentation.experimental.span-suppression-strategy you can enable a different suppression strategy.

For example, if we have a database client which uses Reactor Netty http client which uses Netty networking library, then with the default suppression strategy (semconv), we would have 2 nested spans:

  • CLIENT span with database semantic attributes from the database client instrumentation
  • CLIENT span with http semantic attributes from Reactor Netty instrumentation

With suppression strategy of span-kind, we would have 1 span:

  • CLIENT span with database semantic attributes from the database client instrumentation

And with suppression strategy of none, we would have 3 nested spans:

  • CLIENT span with database semantic attributes from the database client instrumentation
  • CLIENT span with http semantic attributes from Reactor Netty instrumentation
  • CLIENT span with http semantic attributes from Netty instrumentation
Last modified August 6, 2022: Update suppression keys (#1601) (bc6ffc9)