otel java agent unable to export to collector - docker

Question

I'm running JPetStore6 with no code otel java agent and otel collector in the same docker container. Everything starts, but jpet is unable to export data to the collector, despite they are in the same docker container a separate services.

That's the error: Failed to connect to otel-collector/172.19.0.2:4317

docker-compose and otel-collector-config are in the root, the agent is in src/lib folder

docker-compose.yaml:

services:
  jpetstore:
    container_name: jpetstore
    build:
      context: .
    ports:
      - 8080:8080
    restart: always
    environment:
      - JAVA_TOOL_OPTIONS="-javaagent:/opentelemetry-javaagent.jar"
      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:"http://otel-collector:4318/"
      - OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:"http://otel-collector:4318/"
      - OTEL_EXPORTER_OTLP_LOGS_ENDPOINT:"http://otel-collector:4318/"
    volumes:
      - ./src/lib/opentelemetry-javaagent.jar:/opentelemetry-javaagent.jar
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/otel-collector-config.yaml
    expose:
      - "4317"  
      - "4318"
    ports:
      - "4317:4317"  
      - "4318:4318"
    restart: always

otel-collector-config

extensions:
  zpages:
    endpoint: "localhost:55679"
  basicauth/grafana_cloud:
    client_auth:
      username: "MYUSERNAME"
      password: "MYPASSWORD"

receivers:
  otlp:
    protocols:
      grpc:
      http:
  hostmetrics:
    scrapers:
      cpu:
      load:
      memory:
  docker_stats:
    endpoint: unix:///var/run/docker.sock
    collection_interval: 10s

processors:
  batch:
  resourcedetection:
    detectors: ["env", "system"]
    override: false
  transform/drop_unneeded_resource_attributes:
    error_mode: ignore
    trace_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
    metric_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
    log_statements:
      - context: resource
        statements:
          - delete_key(attributes, "k8s.pod.start_time")
          - delete_key(attributes, "os.description")
          - delete_key(attributes, "os.type")
          - delete_key(attributes, "process.command_args")
          - delete_key(attributes, "process.executable.path")
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")
          - delete_key(attributes, "process.runtime.name")
          - delete_key(attributes, "process.runtime.version")
  transform/add_resource_attributes_as_metric_attributes:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["deployment.environment"], resource.attributes["deployment.environment"])
          - set(attributes["service.version"], resource.attributes["service.version"])

exporters:
  otlphttp/grafana_cloud:
    endpoint: "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp"
    auth:
      authenticator: basicauth/grafana_cloud
  debug:
    verbosity: detailed

service:
  extensions: [basicauth/grafana_cloud, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resourcedetection, transform/drop_unneeded_resource_attributes, batch]
      exporters: [otlphttp/grafana_cloud, debug]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [resourcedetection, transform/drop_unneeded_resource_attributes, transform/add_resource_attributes_as_metric_attributes, batch]
      exporters: [otlphttp/grafana_cloud, debug]
    logs:
      receivers: [otlp]
      processors: [resourcedetection, transform/drop_unneeded_resource_attributes, batch]
      exporters: [otlphttp/grafana_cloud, debug]

I supposed it is a problem that the agent is trying to reach localhost, which doesn't exist inside docker, but also after overriding with enviroments still nothing.

Giovanni Zago · Accepted Answer · 2025-02-15 09:51:10Z

I've found the answer.

I needed to find the correct gateway via

docker network inspect "network name" -v

I still don't get the full theory around it, but basically there's not a single gateway from any container to the host machine (or the net, in case you are referring to some www application). But any container has its own network and any network has its gateway.

The solution was to list all networks with

docker network ls

get the container network I'm currently working with (I'm sure there's a better way, but in my case it was just picking the one with the closest name to the container I created) and investigate that network. This is an example result (created by ChatGPT)

[
    {
        "Name": "my_network",
        "Id": "f3d4b7c8e9a01234abcd5678ef90123456789abcde0123456789abcdef012345",
        "Created": "2025-02-15T10:15:30.123456789Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.1.0/24",
                    "Gateway": "192.168.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "c123456789abcdef0123456789abcdef0123456789abcdef0123456789abcd": {
                "Name": "my_container",
                "EndpointID": "e123456789abcdef0123456789abcdef0123456789abcdef0123456789abcd",
                "MacAddress": "02:42:c0:a8:01:02",
                "IPv4Address": "192.168.1.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.name": "br-1234abcd5678",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

and use the gateway specified here "Gateway": "192.168.1.1"

Edit: hope this answer better matches community standards.

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

otel java agent unable to export to collector - docker

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related