feat(opentelemetry source): support OTLP logs, metrics & traces

[pront]

Summary

The opentelemetry source now supports a new decoding mode which can be enabled by setting use_otlp_decoding to true. In this mode,
all events will preserve the OTLP format. These events can be forwarded directly to
the opentelemetry sink without modifications.

A caveat here is that OTLP metrics and Vector metric format differ and thus we treat as logs as they come out the source. These events
cannot be used with existing metrics transforms. However, these can be ingested by the OTEL collectors as metrics.

Vector configuration

Vector Config:

api:
  enabled: true

sources:
  otel:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:5317
    http:
      address: 0.0.0.0:5318
      keepalive:
        max_connection_age_jitter_factor: 0.1
        max_connection_age_secs: 300
    use_oltp_decoding: true

#  internal_metrics:
#    type: internal_metrics
#    scrape_interval_secs: 60


sinks:
  logs_sink:
    type: console
    inputs:
      - otel.logs
    encoding:
      codec: json

  metrics_sink:
    type: console
    inputs:
      - otel.metrics
    encoding:
      codec: json

  traces_sink:
    type: console
    inputs:
      - otel.traces
    encoding:
      codec: json

  otel_sink_logs:
    inputs:
      - otel.logs
    type: opentelemetry
    protocol:
      type: http
      uri: http://localhost:5318/v1/logs
      method: post
      encoding:
        codec: protobuf
        protobuf:
          desc_file: /Users/pavlos.rontidis/CLionProjects/vector/pront/otel/proto/opentelemetry-proto.desc
          message_type: opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest
      framing:
        method: "bytes"
      request:
        headers:
          content-type: "application/x-protobuf"

  otel_sink_metrics:
    inputs:
      - otel.metrics
    type: opentelemetry
    protocol:
      type: http
      uri: http://localhost:5318/v1/metrics
      method: post
      encoding:
        codec: protobuf
        protobuf:
          desc_file: /Users/pavlos.rontidis/CLionProjects/vector/pront/otel/proto/opentelemetry-proto.desc
          message_type: opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest
      framing:
        method: "bytes"
      request:
        headers:
          content-type: "application/x-protobuf"

  otel_sink_traces:
    inputs:
      - otel.traces
    type: opentelemetry
    protocol:
      type: http
      uri: http://localhost:5318/v1/traces
      method: post
      encoding:
        codec: protobuf
        protobuf:
          desc_file: /Users/pavlos.rontidis/CLionProjects/vector/pront/otel/proto/opentelemetry-proto.desc
          message_type: opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest
      framing:
        method: "bytes"
      request:
        headers:
          content-type: "application/x-protobuf"


#  metrics_console_sink:
#    type: console
#    inputs:
#      - internal_metrics
#    encoding:
#      codec: json

How did you test this PR?

Added a new E2E environment that exercises the new code for logs.
Metrics and traces were tested manually. I have these documented and will probably follow up with a docs improvement PR.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

[datadog-vectordotdev]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0980161 | Docs | Was this helpful? Give us feedback!}

[pront]

/ci-run-e2e-opentelemetry-logs

[urseberry]

Thanks for the clarification to my feedback on the release note. I didn't understand the language. I assume "emits treats" is a domain-specific phrase that I'm not familiar with. I'll approve the PR.

[pront]

Thanks for the clarification to my feedback on the release note. I didn't understand the language. I assume "emits treats" is a domain-specific phrase that I'm not familiar with. I'll approve the PR.

This was a typo, my bad. Fixed in 0980161