-
Notifications
You must be signed in to change notification settings - Fork 45
docs: add scorecard integration #94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,105 @@ | ||||||||||||||||||||||
--- | ||||||||||||||||||||||
title: "LLM Observability with Scorecard and OpenLLMetry" | ||||||||||||||||||||||
sidebarTitle: "Scorecard" | ||||||||||||||||||||||
--- | ||||||||||||||||||||||
|
||||||||||||||||||||||
Scorecard is an [AI evaluation and optimization platform](https://www.scorecard.io/) that helps teams build reliable AI systems with comprehensive testing, evaluation, and continuous monitoring capabilities. | ||||||||||||||||||||||
|
||||||||||||||||||||||
## Setup | ||||||||||||||||||||||
|
||||||||||||||||||||||
To integrate OpenLLMetry with Scorecard, you'll need to configure your tracing endpoint and authentication: | ||||||||||||||||||||||
|
||||||||||||||||||||||
### 1. Get your Scorecard API Key | ||||||||||||||||||||||
|
||||||||||||||||||||||
1. Visit your [Settings Page](https://app.scorecard.io/settings) | ||||||||||||||||||||||
2. Copy your API Key | ||||||||||||||||||||||
|
||||||||||||||||||||||
### 2. Configure Environment Variables | ||||||||||||||||||||||
|
||||||||||||||||||||||
```bash | ||||||||||||||||||||||
TRACELOOP_BASE_URL="https://tracing.scorecard.io/otel" | ||||||||||||||||||||||
TRACELOOP_HEADERS="Authorization=Bearer <YOUR_SCORECARD_API_KEY>" | ||||||||||||||||||||||
``` | ||||||||||||||||||||||
|
||||||||||||||||||||||
### 3. Instrument your code | ||||||||||||||||||||||
|
||||||||||||||||||||||
First, install OpenLLMetry and your LLM library: | ||||||||||||||||||||||
|
||||||||||||||||||||||
<CodeGroup> | ||||||||||||||||||||||
```sh Python | ||||||||||||||||||||||
pip install traceloop-sdk openai | ||||||||||||||||||||||
``` | ||||||||||||||||||||||
|
||||||||||||||||||||||
```sh JavaScript | ||||||||||||||||||||||
npm install @traceloop/node-server-sdk openai | ||||||||||||||||||||||
``` | ||||||||||||||||||||||
</CodeGroup> | ||||||||||||||||||||||
|
||||||||||||||||||||||
Then initialize OpenLLMetry and structure your application using workflows and tasks: | ||||||||||||||||||||||
|
||||||||||||||||||||||
<CodeGroup> | ||||||||||||||||||||||
```py Python | ||||||||||||||||||||||
from traceloop.sdk import Traceloop | ||||||||||||||||||||||
from traceloop.sdk.decorators import workflow, task | ||||||||||||||||||||||
from traceloop.sdk.instruments import Instruments | ||||||||||||||||||||||
from openai import OpenAI | ||||||||||||||||||||||
|
||||||||||||||||||||||
# Initialize OpenAI client | ||||||||||||||||||||||
openai_client = OpenAI() | ||||||||||||||||||||||
|
||||||||||||||||||||||
# Initialize OpenLLMetry (reads config from environment variables) | ||||||||||||||||||||||
Traceloop.init(disable_batch=True, instruments={Instruments.OPENAI}) | ||||||||||||||||||||||
|
||||||||||||||||||||||
@workflow(name="simple_chat") | ||||||||||||||||||||||
def simple_workflow(): | ||||||||||||||||||||||
completion = openai_client.chat.completions.create( | ||||||||||||||||||||||
model="gpt-4o-mini", | ||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Typo spotted: The model name "gpt-4o-mini" may contain an extra 'o'. Consider changing it to "gpt-4-mini" if that is the intended model.
Suggested change
|
||||||||||||||||||||||
messages=[{"role": "user", "content": "Tell me a joke"}] | ||||||||||||||||||||||
) | ||||||||||||||||||||||
return completion.choices[0].message.content | ||||||||||||||||||||||
|
||||||||||||||||||||||
# Run the workflow - all LLM calls will be automatically traced | ||||||||||||||||||||||
simple_workflow() | ||||||||||||||||||||||
print("Check Scorecard for traces!") | ||||||||||||||||||||||
``` | ||||||||||||||||||||||
|
||||||||||||||||||||||
```js JavaScript | ||||||||||||||||||||||
import * as traceloop from "@traceloop/node-server-sdk"; | ||||||||||||||||||||||
import OpenAI from "openai"; | ||||||||||||||||||||||
|
||||||||||||||||||||||
// Initialize OpenAI client | ||||||||||||||||||||||
const openai = new OpenAI(); | ||||||||||||||||||||||
|
||||||||||||||||||||||
// Initialize OpenLLMetry with automatic instrumentation | ||||||||||||||||||||||
traceloop.initialize({ | ||||||||||||||||||||||
disableBatch: true, // Ensures immediate trace sending | ||||||||||||||||||||||
instrumentModules: { openAI: OpenAI }, | ||||||||||||||||||||||
}); | ||||||||||||||||||||||
Comment on lines
+73
to
+77
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 Verification agent 🧩 Analysis chainDouble-check The Node SDK expects the module map keys to match the tracer names ( Run a quick smoke test after building the docs example: 🏁 Script executed: # Expect traces to be emitted; zero traces means the key is wrong
node simpleWorkflow.js | grep -i traceloop Length of output: 778 🏁 Script executed: #!/usr/bin/env bash
set -e
# Locate the example workflow script
SCRIPT=$(find . -type f -name "simpleWorkflow.js" | head -n 1)
if [[ -z "$SCRIPT" ]]; then
echo "Error: simpleWorkflow.js not found in repository"
exit 1
fi
echo "Found example at: $SCRIPT"
echo "Running workflow to verify trace output…"
node "$SCRIPT" | grep -i traceloop || echo "⚠️ No traces emitted – please verify that the key in instrumentModules matches the tracer’s registered name (case-sensitive)." Length of output: 203 Ensure correct casing in The module key must exactly match the tracer’s registered name. Update your snippet as follows: // Initialize OpenLLMetry with automatic instrumentation
traceloop.initialize({
disableBatch: true, // Ensures immediate trace sending
- instrumentModules: { openAI: OpenAI },
+ instrumentModules: { OpenAI: OpenAI },
}); If the key casing doesn’t align, instrumentation will be silently disabled. After making this change, please run your example workflow and confirm that traces are emitted as expected. 📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents
|
||||||||||||||||||||||
|
||||||||||||||||||||||
async function simpleWorkflow() { | ||||||||||||||||||||||
return await traceloop.withWorkflow({ name: "simple_chat" }, async () => { | ||||||||||||||||||||||
const completion = await openai.chat.completions.create({ | ||||||||||||||||||||||
model: "gpt-4o-mini", | ||||||||||||||||||||||
messages: [{ role: "user", content: "Tell me a joke" }], | ||||||||||||||||||||||
}); | ||||||||||||||||||||||
return completion.choices[0].message.content; | ||||||||||||||||||||||
}); | ||||||||||||||||||||||
} | ||||||||||||||||||||||
|
||||||||||||||||||||||
# Run the workflow - all LLM calls will be automatically traced | ||||||||||||||||||||||
simpleWorkflow(); | ||||||||||||||||||||||
console.log("Check Scorecard for traces!"); | ||||||||||||||||||||||
``` | ||||||||||||||||||||||
</CodeGroup> | ||||||||||||||||||||||
|
||||||||||||||||||||||
## Features | ||||||||||||||||||||||
|
||||||||||||||||||||||
Once configured, you'll have access to Scorecard's comprehensive observability features: | ||||||||||||||||||||||
|
||||||||||||||||||||||
- **Automatic LLM instrumentation** for popular libraries (OpenAI, Anthropic, etc.) | ||||||||||||||||||||||
- **Structured tracing** with workflows and tasks using `@workflow` and `@task` decorators | ||||||||||||||||||||||
- **Performance monitoring** including latency, token usage, and cost tracking | ||||||||||||||||||||||
- **Real-time evaluation** with continuous monitoring of AI system performance | ||||||||||||||||||||||
- **Production debugging** with detailed trace analysis | ||||||||||||||||||||||
|
||||||||||||||||||||||
For more detailed setup instructions and examples, check out the [Scorecard Tracing Quickstart](https://docs.scorecard.io/intro/tracing-quickstart). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion
Fix malformed code-fence headers.
```sh Python
/```sh JavaScript
are not valid fence identifiers and break syntax highlighting in most Markdown engines.Replace with standard language tags:
If you need a visible label, use the MDX
filename="Python"
attribute instead.🤖 Prompt for AI Agents