Unable to find Datadog logs for checkout API failures

[tanvir362]

I've configured Datadog with opentelemetry-demo by the guide Sending Data from the OpenTelemetry Demo to Datadog. Find my configuration at the end.

Now, the data is being sent to Datadog, but I can't find or identify the logs generated by the checkout API error. I'm using the Search logs (GET) API for fetching logs.

Could you please check if there is anything wrong with my configuration, if I'm using the wrong API, or provide the exact text I should search for?

Here is my otel-collector service config from docker-compose.yml:

  otel-collector:
    image: ${COLLECTOR_CONTRIB_IMAGE}
    container_name: otel-collector
    deploy:
      resources:
        limits:
          memory: 200M
    restart: unless-stopped
    command: [ "--config=/etc/otelcol-config.yml", "--config=/etc/otelcol-config-extras.yml", "--feature-gates=exporter.datadogexporter.UseLogsAgentExporter" ]
    user: 0:0
    volumes:
      - ${HOST_FILESYSTEM}:/hostfs:ro
      - ${DOCKER_SOCK}:/var/run/docker.sock:ro
      - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml
      - ${OTEL_COLLECTOR_CONFIG_EXTRAS}:/etc/otelcol-config-extras.yml
    ports:
      - "${OTEL_COLLECTOR_PORT_GRPC}"
      - "${OTEL_COLLECTOR_PORT_HTTP}"
    depends_on:
      jaeger:
        condition: service_started
      # opensearch:
      #   condition: service_healthy
    logging: *logging
    environment:
      - ENVOY_PORT
      - HOST_FILESYSTEM
      - OTEL_COLLECTOR_HOST
      - OTEL_COLLECTOR_PORT_GRPC
      - OTEL_COLLECTOR_PORT_HTTP
      - DD_SITE_PARAMETER=datadoghq.com
      - DD_API_KEY=<xxxxxxxxxxxxxxx>

And here is how my src/otel-collector/otelcol-config-extras.yml looks:

exporters:
  datadog:
    traces:
      span_name_as_resource_name: true
      trace_buffer: 500
    hostname: "otel-collector-docker"
    api:
      site: ${env:DD_SITE_PARAMETER}
      key: ${env:DD_API_KEY}

processors:
  resource:
    attributes:
      - key: deployment.environment
        value: "otel"
        action: upsert

connectors:
  datadog/connector:
    traces:
      span_name_as_resource_name: true

service:
  pipelines:
    traces:
      processors: [resource, batch]
      exporters: [otlp, debug, spanmetrics, datadog, datadog/connector]
    metrics:
      receivers: [docker_stats, httpcheck/frontendproxy, otlp, prometheus, redis, spanmetrics, datadog/connector]
      processors: [resource, batch]
      exporters: [otlphttp/prometheus, debug, datadog]
    logs:
      processors: [resource, batch]
      exporters: [opensearch, debug, datadog]

[julianocosta89]

hey @tanvir362 👋🏽

Juliano here, a maintainer of the Demo.

There are a couple of things here that we need to discuss 😅

First and more important, please rotate your DD_API_KEY as you have just shared it publicly.

Also, this PR should be opened on the Datadog fork, but I work at Datadog, and I can help you out.

I'm not sure what do you mean by checkout API error.
Could you elaborate a bit further?

Just one thing to also highlight, is that checkout service doesn't have OTLP logs in place yet, meaning that this service is not sending it's logs anywhere ATM.
We are keeping track of the implementation on this issue: #1472

[tanvir362]

Hi Juliano 👋

Thanks for the detailed response. I've rotated the DD_API_KEY, and really appreciate the heads-up. When I mentioned the checkout API error, I was referring to the internal server error from the /api/checkout endpoint that occurs when placing an order. We expected to see those logs in Datadog, but later realized they were not being sent at all. Your note about the checkout service not having OTLP logs confirms this.

We have since figured out a workaround and managed to make it work.

[julianocosta89]

Awesome to hear @tanvir362 🤩

BTW, @kerneljack recently added OTLP logs to product-catalog (another Go service in the Demo) #2285.
His plan was to also add OTLP to checkout. @kerneljack have you started working on it?
If not, I'd suggest @tanvir362 maybe contributing the changes they have made 😊

[kerneljack]

Hi @julianocosta89 - yes I started looking into it a couple days ago, my plan was to do some more work on it over the next few days and hopefully have a PR ready by the weekend.

[julianocosta89]

awesome!

thank you for that 😊