Skip to content

docs: add scorecard integration #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@
"openllmetry/integrations/newrelic",
"openllmetry/integrations/otel-collector",
"openllmetry/integrations/oraclecloud",
"openllmetry/integrations/scorecard",
"openllmetry/integrations/service-now",
"openllmetry/integrations/signoz",
"openllmetry/integrations/sentry",
Expand Down Expand Up @@ -183,9 +184,7 @@
},
{
"group": "Costs",
"pages": [
"api-reference/costs/property_costs"
]
"pages": ["api-reference/costs/property_costs"]
}
],
"redirects": [
Expand Down
1 change: 1 addition & 0 deletions openllmetry/integrations/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ in any observability platform that supports OpenTelemetry.
title="Oracle Cloud"
href="/openllmetry/integrations/oraclecloud"
></Card>
<Card title="Scorecard" href="/openllmetry/integrations/scorecard"></Card>
<Card
title="Service Now Cloud Observability"
href="/openllmetry/integrations/service-now"
Expand Down
105 changes: 105 additions & 0 deletions openllmetry/integrations/scorecard.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
---
title: "LLM Observability with Scorecard and OpenLLMetry"
sidebarTitle: "Scorecard"
---

Scorecard is an [AI evaluation and optimization platform](https://www.scorecard.io/) that helps teams build reliable AI systems with comprehensive testing, evaluation, and continuous monitoring capabilities.

## Setup

To integrate OpenLLMetry with Scorecard, you'll need to configure your tracing endpoint and authentication:

### 1. Get your Scorecard API Key

1. Visit your [Settings Page](https://app.scorecard.io/settings)
2. Copy your API Key

### 2. Configure Environment Variables

```bash
TRACELOOP_BASE_URL="https://tracing.scorecard.io/otel"
TRACELOOP_HEADERS="Authorization=Bearer <YOUR_SCORECARD_API_KEY>"
```

### 3. Instrument your code

First, install OpenLLMetry and your LLM library:

<CodeGroup>
```sh Python
pip install traceloop-sdk openai
```

```sh JavaScript
npm install @traceloop/node-server-sdk openai
```
</CodeGroup>
Comment on lines +29 to +36
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Fix malformed code-fence headers.

```sh Python / ```sh JavaScript are not valid fence identifiers and break syntax highlighting in most Markdown engines.
Replace with standard language tags:

-```sh Python
+```bash-```sh JavaScript
+```bash

If you need a visible label, use the MDX filename="Python" attribute instead.

🤖 Prompt for AI Agents
In openllmetry/integrations/scorecard.mdx around lines 29 to 36, the code-fence
headers use invalid identifiers like ```sh Python and ```sh JavaScript, which
break syntax highlighting. Replace these with standard language tags such as
```bash for both code blocks. If you want to label the code blocks by language,
use the MDX filename attribute instead of combining multiple identifiers in the
fence header.


Then initialize OpenLLMetry and structure your application using workflows and tasks:

<CodeGroup>
```py Python
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, task
from traceloop.sdk.instruments import Instruments
from openai import OpenAI

# Initialize OpenAI client
openai_client = OpenAI()

# Initialize OpenLLMetry (reads config from environment variables)
Traceloop.init(disable_batch=True, instruments={Instruments.OPENAI})

@workflow(name="simple_chat")
def simple_workflow():
completion = openai_client.chat.completions.create(
model="gpt-4o-mini",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo spotted: The model name "gpt-4o-mini" may contain an extra 'o'. Consider changing it to "gpt-4-mini" if that is the intended model.

Suggested change
model="gpt-4o-mini",
model="gpt-4-mini",

messages=[{"role": "user", "content": "Tell me a joke"}]
)
return completion.choices[0].message.content

# Run the workflow - all LLM calls will be automatically traced
simple_workflow()
print("Check Scorecard for traces!")
```

```js JavaScript
import * as traceloop from "@traceloop/node-server-sdk";
import OpenAI from "openai";

// Initialize OpenAI client
const openai = new OpenAI();

// Initialize OpenLLMetry with automatic instrumentation
traceloop.initialize({
disableBatch: true, // Ensures immediate trace sending
instrumentModules: { openAI: OpenAI },
});
Comment on lines +73 to +77
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Double-check instrumentModules key casing.

The Node SDK expects the module map keys to match the tracer names (openAI vs OpenAI). Mismatching case silently disables instrumentation.

Run a quick smoke test after building the docs example:


🏁 Script executed:

# Expect traces to be emitted; zero traces means the key is wrong
node simpleWorkflow.js | grep -i traceloop

Length of output: 778


🏁 Script executed:

#!/usr/bin/env bash
set -e
# Locate the example workflow script
SCRIPT=$(find . -type f -name "simpleWorkflow.js" | head -n 1)
if [[ -z "$SCRIPT" ]]; then
  echo "Error: simpleWorkflow.js not found in repository"
  exit 1
fi
echo "Found example at: $SCRIPT"
echo "Running workflow to verify trace output…"
node "$SCRIPT" | grep -i traceloop || echo "⚠️ No traces emitted – please verify that the key in instrumentModules matches the tracer’s registered name (case-sensitive)."

Length of output: 203


Ensure correct casing in instrumentModules

The module key must exactly match the tracer’s registered name. Update your snippet as follows:

// Initialize OpenLLMetry with automatic instrumentation
traceloop.initialize({ 
    disableBatch: true,  // Ensures immediate trace sending
-   instrumentModules: { openAI: OpenAI },
+   instrumentModules: { OpenAI: OpenAI },
});

If the key casing doesn’t align, instrumentation will be silently disabled. After making this change, please run your example workflow and confirm that traces are emitted as expected.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Initialize OpenLLMetry with automatic instrumentation
traceloop.initialize({
disableBatch: true, // Ensures immediate trace sending
instrumentModules: { openAI: OpenAI },
});
// Initialize OpenLLMetry with automatic instrumentation
traceloop.initialize({
disableBatch: true, // Ensures immediate trace sending
instrumentModules: { OpenAI: OpenAI },
});
🤖 Prompt for AI Agents
In openllmetry/integrations/scorecard.mdx around lines 73 to 77, the key used in
the instrumentModules object must exactly match the tracer's registered name
with correct casing. Change the key from "openAI" to the exact registered name
with proper casing to ensure instrumentation is enabled. After updating the key,
run the example workflow to verify that traces are emitted correctly.


async function simpleWorkflow() {
return await traceloop.withWorkflow({ name: "simple_chat" }, async () => {
const completion = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Tell me a joke" }],
});
return completion.choices[0].message.content;
});
}

# Run the workflow - all LLM calls will be automatically traced
simpleWorkflow();
console.log("Check Scorecard for traces!");
```
</CodeGroup>

## Features

Once configured, you'll have access to Scorecard's comprehensive observability features:

- **Automatic LLM instrumentation** for popular libraries (OpenAI, Anthropic, etc.)
- **Structured tracing** with workflows and tasks using `@workflow` and `@task` decorators
- **Performance monitoring** including latency, token usage, and cost tracking
- **Real-time evaluation** with continuous monitoring of AI system performance
- **Production debugging** with detailed trace analysis

For more detailed setup instructions and examples, check out the [Scorecard Tracing Quickstart](https://docs.scorecard.io/intro/tracing-quickstart).