Implement OpenLLMetry Tracing in Python Application

[nirga]

This pull request implements OpenLLMetry tracing in our Python application to enable visibility into our LLM calls through Traceloop.

Changes made:

1. Added openllmetry dependency to pyproject.toml
2. Created a new file src/prompt_optimizer/tracing.py to initialize the OpenLLMetry tracer
3. Modified src/prompt_optimizer/main.py to add tracing to the main flow
4. Updated src/prompt_optimizer/evaluate_crew/evaluate_crew.py and src/prompt_optimizer/optimize_crew/optimize_crew.py to import the tracer

Key benefits:

• Improved observability of our LLM operations
• Ability to monitor and analyze performance of our prompt optimization flow
• Enhanced debugging capabilities for our AI-driven processes

To complete the setup, please add your Traceloop API key to the environment variables as OPENLLMETRY_API_KEY.

Testing:

• Run the application and verify that traces are being sent to your Traceloop dashboard.
• Check that the added instrumentation doesn't impact the overall performance of the application.

Note: Make sure to keep your API key secure and never commit it directly to the repository.

---

Important

Implements OpenLLMetry tracing for enhanced observability of LLM operations, adding a new dependency and modifying main flow for tracing.

• Dependencies:
  • Added openllmetry to pyproject.toml.
• Tracing Initialization:
  • Created tracing.py to initialize OpenLLMetry tracer.
  • Imports tracer_instance in evaluate_crew.py, main.py, and optimize_crew.py.
• Main Flow Modifications:
  • Added tracing to evaluate_prompt() and optimize_prompt() in main.py.
  • Traces include attributes like prompt, score, retry_count, and failure_reasons.
• Documentation:
  • Added TRACING.md for setup and usage instructions.
• Environment:
  • Requires TRACELOOP_API_KEY for Traceloop integration, optional for local development.

^{This description was created by for 6a11846. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Caution

Changes requested ❌

Reviewed everything up to 6a11846 in 1 minute and 35 seconds. Click for details.

• Reviewed 258 lines of code in 6 files
• Skipped 0 files when reviewing.
• Skipped posting 6 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. TRACING.md:54

• Draft comment:
  Add a newline at the end of the file for consistency and improved tooling support.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

2. pyproject.toml:16

• Draft comment:
  Verify that 'openllmetry' is the intended dependency name (it resembles 'opentelemetry').
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% The comment violates several rules: 1) It asks for verification rather than stating a clear issue 2) It's speculative - just because names are similar doesn't mean it's wrong 3) We have no strong evidence that openllmetry is incorrect - it could be a valid package name. 4) Dependency choices should be trusted unless we have clear evidence of an error. What if openllmetry is actually a typo and this comment could prevent a real issue? What if opentelemetry is indeed the intended package? Without concrete evidence that this is wrong, we should trust the author's dependency choice. If it was a typo, it would fail during build/install anyway. Delete this comment as it violates multiple rules - it's speculative, asks for verification, and comments on dependencies without clear evidence of an error.

3. src/prompt_optimizer/main.py:35

• Draft comment:
  Consider using a proper logging framework instead of 'print' for better production log management.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

4. src/prompt_optimizer/optimize_crew/optimize_crew.py:6

• Draft comment:
  The 'tracer_instance' import is not used in this file. Remove it if it's not needed to keep the code clean.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% While the comment is technically correct that the import appears unused, unused imports are typically caught by linters and IDE tools. This falls under the rule about not commenting on things that would be caught by the build. Most Python builds include checks for unused imports. Maybe this codebase doesn't have linting set up properly? Maybe the import is needed for some side effect or will be used in code that's coming soon? Even if linting isn't set up, commenting on unused imports is still not a high-value review comment. And if the import is needed for future code, that's speculative. Delete this comment as it violates the rule about not commenting on things that would be caught by the build.

5. src/prompt_optimizer/tracing.py:10

• Draft comment:
  Consider replacing 'print' with a logging library (e.g., logging.warning) for better log management and production readiness.
• Reason this comment was not posted:
  Confidence changes required: 33% <= threshold 50% None

6. pyproject.toml:16

• Draft comment:
  Typo: The dependency 'openllmetry>=0.27.0' appears to have a typographical error. Did you mean 'opentelemetry>=0.27.0'?
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% I need to be careful here. While 'opentelemetry' is a well-known package, 'openllmetry' could be a legitimate package name, possibly a specialized version for LLM telemetry. Without being able to verify the package's existence or the author's intention, making assumptions about typos could be misleading. This violates the rule about not making speculative comments or asking for confirmation. I might be overly cautious - if openllmetry is truly a typo, this comment could prevent a real issue. The similarity to 'opentelemetry' is striking. However, the rules clearly state we need STRONG EVIDENCE to keep a comment. Without being able to verify if openllmetry is a real package or not, we don't have strong evidence this is actually a typo. The comment should be deleted as it's speculative and asks for confirmation without strong evidence of being incorrect.

Workflow ID: wflow_osfgH7qIsbAXIiKA

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

The imported 'tracer_instance' is not used in this module. Consider removing it or adding instrumentation if needed.

Suggested change

from prompt_optimizer.tracing import tracer_instance